phase: 191_source_name_labels_tags
All gates re-verified against a stable tree. Report: **Verified (all 6 tasks in `complete/`; this was the final pass)** - `SourceLabel`+alembic `0030` (dev DB `alembic current` = `0030 (head)`, table present, 0 rows); admin-only `GET/PUT/DELETE /api/source-labels` (`require_admin` router-wide, `resolve_source_name`, partial-update caps/404s); Sources-page "Names, labels and tags" card (read-only derived-name `<code>`, per-row `saveInFlight`, 404 note that survives the settle, chips reuse `.git-source-hidden-count`); `question_name_hits` D2 rule; ONE `all_labels` read per turn in `chat.py` feeding both the `source_name_hit` flag and the prompt surfaces; `source_name_hit_bonus=0.005` in `_selection_order`; ` name=`/` label=` in `build_high_prompt` + `build_overview_prompt(names=…)`. - **Completion criteria:** E2E `tests/e2e/test_source_name_labels.py` ✅ 7 passed (arms a–e: editor all kinds, control/named question, rename-never-breaks-identity, 404 note, reset) · full suite ✅ · coverage ✅ · README/AGENTS ✅ · phase dir in `complete/` ✅ · commit left to the harness. - **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 7160 stmts / 16 missed = 99%** (>90%) · `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` → **7 passed, exit 0** · regression E2E isolated, exit 0: `test_document_viewer`, `test_admin_auth`, `test_api_tokens`, `test_git_sources_admin`, `test_kb_tree`, `test_local_directory_sources` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors. - **Battery (task 04 criterion):** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` at default vs `BOR_SOURCE_NAME_HIT_BONUS=0` → **`diff` empty (line-for-line identical)**, matching the recorded arms (dev KB is empty, so every question deflects; the observable boost is unit/E2E-pinned). - **Documentation:** README — fixed the `/git-sources.html` rows (nav label is **Sources**) and cross-linked the phase-191 card from that section; full audit otherwise clean: all 86 `BOR_*` settings documented with defaults matching `app/config.py`, 0 broken file/dir references and 0 broken anchors in README+AGENTS, AGENTS rule 17 re-checked line-by-line against the code. - **Notable / deviations:** (1) a **concurrent pipeline process** (`phased-execution/progress.mjs` for `191_…_00_phase`) was mutating this same working tree mid-pass — it rewrote `tests/e2e/test_source_name_labels.py` (11→7 tests), `frontend/assets/git-sources.js`, `tests/unit/test_source_labels_editor_ui.py` and `AGENTS.md` at 05:27; every gate above was (re-)run after those edits with the tree no source file newer than the green full run. (2) The task-06 report/phase text claims regression suites `test_chat_token_gate.py`, `test_git_sources_page.py`, `test_rag_browser.py` are green — **those files have never existed in git history** (those runs exit 4); I ran the real equivalents listed above instead (phase files are not mine to edit). (3) **No live changes**: the VM DB already has `source_labels` (0 rows) but the deployed app predates this uncommitted code, so deploying `/api/source-labels` to the live service is the owner's post-commit step, not part of this phase's criteria. - **Next pending phase:** none — `191_source_name_labels_tags` is the only entry in `.agents/phases/todo/`.
This commit is contained in:
+51
-1
@@ -258,7 +258,7 @@ import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
from collections.abc import AsyncIterator, Mapping, Sequence
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -301,6 +301,11 @@ from app.rag.retriever import (
|
||||
weak_hit_titles,
|
||||
)
|
||||
from app.rag.scaffolding import ScaffoldingFilter # phase 71: the streaming filter
|
||||
from app.rag.source_labels import (
|
||||
SourceLabelView,
|
||||
all_labels,
|
||||
question_name_hits,
|
||||
)
|
||||
from app.rag.suggestions import derive_suggestions
|
||||
from app.rag.summarizer import (
|
||||
IMAGE_FALLBACK_MIME,
|
||||
@@ -461,6 +466,7 @@ def plan_turn(
|
||||
notes: Sequence[str] | None = None,
|
||||
kb_overview: str | None = None,
|
||||
folder_lines: Sequence[str] = (),
|
||||
names: Mapping[str, SourceLabelView] | None = None,
|
||||
) -> TurnPlan:
|
||||
"""Apply the honesty gate (A8, revised) and assemble prompt + context.
|
||||
|
||||
@@ -502,6 +508,15 @@ def plan_turn(
|
||||
phase-118 shape. The :class:`TurnPlan`, the durable record, and the
|
||||
per-turn log line are untouched.
|
||||
|
||||
*names* (phase 191, D4): the source-name → ``SourceLabelView`` map of
|
||||
the owner's label rows — the SAME single ``all_labels`` read the
|
||||
endpoint performs for the task-04 selection boost (ONE query per
|
||||
turn). The HIGH branch hands it to ``build_high_prompt`` (the
|
||||
additive ``name`` / ``label`` attributes of the ``<document>``
|
||||
blocks); the LOW branch IGNORES it — the deflected prompt stays
|
||||
byte-identical. ``None`` ⇒ every block is byte-identical to
|
||||
pre-phase.
|
||||
|
||||
``summary_hits`` (phase 30; phase 118 redefinition) counts the hit
|
||||
chunks with ``is_summary`` whose parent document is in the
|
||||
SUGGESTED set — both the HIGH and the LOW branch record it.
|
||||
@@ -561,6 +576,10 @@ def plan_turn(
|
||||
# byte-identical to pre-phase (no clause, and agent_tools
|
||||
# offers four tools, not five).
|
||||
show_text=settings.answer_text,
|
||||
# Phase 191 (D4, task 05): the owner's source names ride the
|
||||
# same map the boost flags came from — one all_labels read
|
||||
# per turn, no second label query here.
|
||||
names=names,
|
||||
),
|
||||
suggested,
|
||||
related_docs,
|
||||
@@ -860,6 +879,33 @@ async def chat(
|
||||
kb_overview = load_kb_overview(step_db)
|
||||
with SessionLocal() as step_db:
|
||||
chunks = retrieve(step_db, request.message, question_vec)
|
||||
# Phase 191 (D3 task 04 + D4 task 05): ONE
|
||||
# all_labels read per turn feeds BOTH surfaces —
|
||||
# the source-name hit flags here AND the prompt
|
||||
# surfaces (the map rides into plan_turn →
|
||||
# build_high_prompt's name/label attributes). The
|
||||
# hit set is the pure D2 rule per view, then a
|
||||
# loop over the few dozen retrieved chunks against
|
||||
# the handful of hit sources (never a per-chunk
|
||||
# query — task 01's question_name_hit_sources is
|
||||
# this same pair, inlined here so the read is
|
||||
# shared). The flags ride the chunk objects into
|
||||
# EVERY selection call below (plan_turn's walk
|
||||
# included); the source_name_hit_bonus itself is
|
||||
# pulled from settings inside the selection
|
||||
# helpers. No label rows → empty map → no flags
|
||||
# and no name attributes → every surface is the
|
||||
# pre-phase one; the per-turn log line is
|
||||
# untouched (the boost is invisible to it).
|
||||
labels = all_labels(step_db)
|
||||
source_name_hits = {
|
||||
view.source
|
||||
for view in labels.values()
|
||||
if question_name_hits(view, request.message)
|
||||
}
|
||||
for rc in chunks:
|
||||
if rc.document.source in source_name_hits:
|
||||
rc.source_name_hit = True
|
||||
# Phase 119 (D3, LOCKED A4): the suggested-folder
|
||||
# context — the SAME deterministic suggested walk
|
||||
# plan_turn performs internally (one extra cheap
|
||||
@@ -880,6 +926,10 @@ async def chat(
|
||||
notes=steering_notes,
|
||||
kb_overview=kb_overview,
|
||||
folder_lines=folder_lines,
|
||||
# Phase 191 (D4, task 05): the SAME single read —
|
||||
# the HIGH prompt learns the name→docs mapping
|
||||
# from these views (no second label query).
|
||||
names=labels,
|
||||
)
|
||||
except Exception: # noqa: BLE001 — DB failure mid-turn
|
||||
logger.exception(
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
"""Source name/label/tags admin API (phase 191, task 02).
|
||||
|
||||
``TODO.md`` L4: "Sources need a name, a label, tag(s) so you can focus
|
||||
the model when prompting if you have a project name or a standard name
|
||||
you reference things by." Task 01 stored that (the ``source_labels``
|
||||
table + :mod:`app.rag.source_labels`); this module is its whole HTTP
|
||||
surface — the API the Sources page's per-source editor (task 03)
|
||||
drives, admin-gated like the phase-91 Theme tab and the phase-168
|
||||
Settings panel.
|
||||
|
||||
Routes (under ``/api`` via the ``main`` registration — the house prefix
|
||||
convention, the :mod:`app.api.ui_settings` /
|
||||
:mod:`app.api.admin_settings` style, so the wire paths are exactly the
|
||||
D5 ones):
|
||||
|
||||
* ``GET /api/source-labels`` — the FULL source catalogue: (a) every
|
||||
``git_sources`` row's DERIVED source name (git AND local rows —
|
||||
:func:`app.rag.source_removal.resolve_source_name`, the one helper
|
||||
that mirrors the import pipeline's naming for both kinds — and, while
|
||||
the table is empty, the ``BOR_GIT_SOURCES`` env names, which
|
||||
:func:`app.rag.git_sources.effective_sources` already surfaces as
|
||||
synthetic git rows), plus (b) every indexed document source no row
|
||||
derives (the uploads — ONE ``SELECT DISTINCT source``, no ``content``,
|
||||
the phase-126 load discipline). De-duplicated (the phase-69 sibling
|
||||
case: two rows, one name) and sorted by ``source``, so the page and
|
||||
the phase's E2E always see the same order. Each entry carries its
|
||||
label row or the ``null``/``[]`` defaults — one
|
||||
:func:`app.rag.source_labels.all_labels` read for the whole list,
|
||||
never a query per source. Reads only; creates nothing.
|
||||
* ``PUT /api/source-labels/{source}`` — the upsert. The path param is
|
||||
capped at the ``documents.source`` 120 (the framework's 422 above
|
||||
it); a ``source`` matching NO registry row AND NO indexed document is
|
||||
a 404 BEFORE anything is written (D5: no orphan label rows — a label
|
||||
for a source that exists nowhere would be a dead boost input, and
|
||||
nothing in the UI could ever list it). The body is the partial-update
|
||||
shape (the phase-168 precedent): an omitted field keeps its stored
|
||||
value, an explicit ``null`` clears it, a value sets it. The tags go
|
||||
through :func:`app.rag.source_labels.normalize_tags` — the ONE tag
|
||||
rule (trim, case-insensitive de-dupe, first case wins) — and its
|
||||
``ValueError`` becomes the framework's 422, NEVER a 500. The router
|
||||
owns the session here (the phase-91 commit convention): ``set_label``
|
||||
flushes, this route commits, and a rejected request writes NOTHING.
|
||||
200 → the stored :class:`SourceLabelOut`.
|
||||
* ``DELETE /api/source-labels/{source}`` — the reset: the row goes, 404
|
||||
when there is no row to remove. Like the phase-168
|
||||
``DELETE /api/admin/settings/{key}`` the 200 body is the entry's
|
||||
post-reset state — the same shape with the fields cleared — so the
|
||||
row re-renders from the response it already has.
|
||||
|
||||
One invariant the routes exist to protect (D1, owner-confirmed): a
|
||||
label row is an ALIAS. ``Document.source``, the ``source/path`` document
|
||||
identity, the read/grep/``show_text`` locators, the citations and the KB
|
||||
tree are ALL untouched by anything here — calling "gitea-forge" "the git
|
||||
server" changes only what the lite model and the chat model are TOLD
|
||||
(tasks 04/05), never what a saved chat, a shared page or a verbatim
|
||||
panel points at. That is why ``source`` is a path param that is LOOKED
|
||||
UP and never rewritten, and why the catalogue's identity column stays
|
||||
the derived name.
|
||||
|
||||
Logging (AGENTS rule 10): one INFO line per WRITE, carrying the action
|
||||
and the source, never the values (the ``admin_settings`` discipline —
|
||||
the owner's text belongs in the row, not in ``server.log``). GETs log
|
||||
nothing here: :class:`app.core.response_budget.ResponseBudgetMiddleware`
|
||||
already writes the per-response ``api method=… bytes=…`` line for every
|
||||
``/api/*`` route.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Annotated, Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Path
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.auth import require_admin
|
||||
from app.db import get_db
|
||||
from app.models import Document
|
||||
from app.rag.git_sources import effective_sources
|
||||
from app.rag.source_labels import (
|
||||
SourceLabelView,
|
||||
all_labels,
|
||||
delete_label,
|
||||
normalize_tags,
|
||||
set_label,
|
||||
)
|
||||
from app.rag.source_removal import resolve_source_name
|
||||
from app.schemas import SourceLabelList, SourceLabelOut, SourceLabelUpdate
|
||||
|
||||
logger = logging.getLogger("app.api.source_labels")
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/source-labels",
|
||||
tags=["source-labels"],
|
||||
dependencies=[Depends(require_admin)], # phase 191: the owner's naming surface
|
||||
)
|
||||
|
||||
|
||||
def _catalogue_names(db: Session) -> list[str]:
|
||||
"""Every source the owner can label, de-duplicated and sorted.
|
||||
|
||||
Exactly the two reads the D5 catalogue is made of — no third:
|
||||
|
||||
(a) the registry: :func:`app.rag.git_sources.effective_sources` —
|
||||
the ``git_sources`` rows of BOTH kinds in ``(added_at, id)``
|
||||
order, or the synthetic env rows while the table is empty —
|
||||
named by :func:`app.rag.source_removal.resolve_source_name`, the
|
||||
single helper that spells a row's name the way the importer
|
||||
stores it in ``Document.source`` (``repo_name`` of the URL for
|
||||
git rows, the expanded path's basename for local rows). Reused,
|
||||
never re-derived: an expression copied here would drift from the
|
||||
importer and the label rows would key nothing.
|
||||
(b) the indexed documents: ``SELECT DISTINCT source`` — one query
|
||||
over the indexed corpus (the column only, never the entity: no
|
||||
``content``, phase-126/185 discipline), so an upload — or any
|
||||
document whose registry row is gone — still lists and stays
|
||||
labelable.
|
||||
|
||||
Sorted because the page renders this order and the phase's E2E pins
|
||||
it; the set collapses the phase-69 sibling case (two rows resolving
|
||||
to one name) to a single entry.
|
||||
"""
|
||||
rows, _origin = effective_sources(db)
|
||||
names: set[str] = {resolve_source_name(row) for row in rows}
|
||||
names.update(db.scalars(select(Document.source).distinct()).all())
|
||||
return sorted(names)
|
||||
|
||||
|
||||
def _shape(source: str, view: SourceLabelView | None) -> SourceLabelOut:
|
||||
"""One entry of the catalogue: the label row's values, or the
|
||||
``null``/``[]`` defaults when the source has no row (the pre-phase
|
||||
state — indistinguishable from a cleared label, and that is the
|
||||
point: D1 says a label is data beside the name, never a rename)."""
|
||||
return SourceLabelOut(
|
||||
source=source,
|
||||
display_name=view.display_name if view else None,
|
||||
label=view.label if view else None,
|
||||
tags=list(view.tags) if view else [],
|
||||
)
|
||||
|
||||
|
||||
@router.get("", response_model=SourceLabelList)
|
||||
def list_source_labels(
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> SourceLabelList:
|
||||
"""The catalogue: every source of every kind, each with its label or
|
||||
the null defaults, sorted by ``source``.
|
||||
|
||||
Two reads for the names (:func:`_catalogue_names`) plus ONE for all
|
||||
the label rows — the table carries a handful of rows, so the list is
|
||||
assembled in Python instead of issuing a lookup per source (the
|
||||
phase-119 one-query convention).
|
||||
|
||||
The catalogue is the SOURCE list, not the label list: a label row
|
||||
whose source has since lost both its registry row and its documents
|
||||
does not appear here (it can boost nothing — the D2 match runs
|
||||
against retrieved chunks' ``document.source``), and it stays
|
||||
clearable through ``DELETE`` below, which never consults the
|
||||
registry.
|
||||
"""
|
||||
labels = all_labels(db)
|
||||
return SourceLabelList(
|
||||
labels=[_shape(name, labels.get(name)) for name in _catalogue_names(db)]
|
||||
)
|
||||
|
||||
|
||||
@router.put("/{source}", response_model=SourceLabelOut)
|
||||
def put_source_label(
|
||||
source: Annotated[str, Path(max_length=120)],
|
||||
payload: SourceLabelUpdate,
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> SourceLabelOut:
|
||||
"""Set or clear one source's name/label/tags (200 → the stored row).
|
||||
|
||||
The order is the D5 contract:
|
||||
|
||||
1. **404 first** — *source* matches no registry row name and no
|
||||
indexed document (the same two reads the GET makes): a label for
|
||||
a source that exists nowhere is refused and nothing is written.
|
||||
A typo'd name would otherwise sit in the table with no row to
|
||||
join, no documents to boost and nothing in the UI to list it.
|
||||
2. **Then the tags** — :func:`~app.rag.source_labels.normalize_tags`
|
||||
(trim, drop the entries that are empty afterwards, de-duplicate
|
||||
case-insensitively with the first case kept); its ``ValueError``
|
||||
becomes a 422 carrying its own fixed, input-free reason. The
|
||||
request shape caps the same numbers, so today the raise is the
|
||||
belt — the pin is that the belt can never fire as a 500.
|
||||
3. **Then the write** — only the fields the request actually carried
|
||||
(``model_fields_set`` — the phase-168 partial-update shape:
|
||||
omitted keeps, ``null`` clears, a value sets) reach
|
||||
:func:`~app.rag.source_labels.set_label`, and the commit happens
|
||||
here (the router owns the session, the phase-91 convention). A
|
||||
rejection at step 1 or 2 leaves any existing row byte-identical.
|
||||
|
||||
An all-omitted body is therefore a legitimate 200 no-op (the
|
||||
``GitSourcePatchIn`` "all absent → 200 no-op" precedent): the row is
|
||||
touched and reported, nothing changes.
|
||||
"""
|
||||
if source not in _catalogue_names(db):
|
||||
raise HTTPException(status_code=404, detail="source not found")
|
||||
|
||||
write: dict[str, Any] = {}
|
||||
carried = payload.model_fields_set
|
||||
if "display_name" in carried:
|
||||
write["display_name"] = payload.display_name
|
||||
if "label" in carried:
|
||||
write["label"] = payload.label
|
||||
if "tags" in carried:
|
||||
try:
|
||||
write["tags"] = normalize_tags(payload.tags or [])
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
|
||||
row = set_label(db, source, **write)
|
||||
db.commit()
|
||||
# The action and the key only — never the owner's text (AGENTS rule
|
||||
# 10, the ``admin_settings`` write-line discipline).
|
||||
logger.info(
|
||||
"source labels action=set source=%s fields=%s",
|
||||
source,
|
||||
",".join(sorted(write)) or "none",
|
||||
)
|
||||
return SourceLabelOut(
|
||||
source=row.source,
|
||||
display_name=row.display_name,
|
||||
label=row.label,
|
||||
tags=list(row.tags or []),
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{source}", response_model=SourceLabelOut)
|
||||
def delete_source_label(
|
||||
source: Annotated[str, Path(max_length=120)],
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> SourceLabelOut:
|
||||
"""Reset one source — the row goes and the next turn sees the
|
||||
pre-phase state.
|
||||
|
||||
404 (one fixed detail, ``source label not found``) when there is no
|
||||
label row for it. Unlike the PUT this does NOT consult the registry
|
||||
or the documents: the reset has to work for a row whose source has
|
||||
since been pruned or whose registry row was deleted (phase 69's
|
||||
total removal), or the label would be un-clearable by construction.
|
||||
"""
|
||||
if not delete_label(db, source):
|
||||
raise HTTPException(status_code=404, detail="source label not found")
|
||||
db.commit()
|
||||
logger.info("source labels action=clear source=%s", source)
|
||||
return SourceLabelOut(source=source, display_name=None, label=None, tags=[])
|
||||
@@ -430,6 +430,23 @@ class Settings(BaseSettings):
|
||||
#: records the realized margins).
|
||||
name_hit_bonus: float = 0.005
|
||||
|
||||
#: (phase 191, D3, LOCKED A3): the SECOND, independent selection-time
|
||||
#: bonus — a document whose SOURCE is named by the question (the D2
|
||||
#: rule over the ``source_labels`` row: a question token equals, in
|
||||
#: normalized form, a token of the source's display_name / label /
|
||||
#: tags) gets this added to its best fused chunk score in the SAME
|
||||
#: shared selection walk as the phase-119 ``name_hit_bonus`` (a
|
||||
#: separate knob: the two signals stay independently kill-able).
|
||||
#: Default 0.005 = the phase-119 starting point; ``0`` = off (the
|
||||
#: walk returns byte-identical to pre-phase — the kill switch); a
|
||||
#: negative value fails startup loudly. Bounded per DOCUMENT, applied
|
||||
#: ONCE, at most TWO bonuses on the fused scale (name_hit + source
|
||||
#: name_hit = 0.010 maximum head start — no amplification); the
|
||||
#: bonus lives in the SELECTION layer only — ``score`` / ``cosine`` /
|
||||
#: ``fts_hit`` / ``query_log.top_score`` are untouched (the
|
||||
#: phase-119 LOCKED A3 clause, mirrored).
|
||||
source_name_hit_bonus: float = 0.005
|
||||
|
||||
# --- Admin & sign-in (phase 16; A10 revised 2026-08-22) ---
|
||||
# Single-admin auth via a signed session cookie (Starlette
|
||||
# SessionMiddleware — no new services, no DB tables). Both secrets are
|
||||
@@ -972,6 +989,17 @@ class Settings(BaseSettings):
|
||||
raise ValueError("name_hit_bonus must be >= 0 (0 = off)")
|
||||
return v
|
||||
|
||||
@field_validator("source_name_hit_bonus")
|
||||
@classmethod
|
||||
def _source_name_hit_bonus_non_negative(cls, v: float) -> float:
|
||||
"""``0`` is the byte-identical kill switch (the pre-phase-191
|
||||
selection order) — a NEGATIVE bonus would demote source-named
|
||||
documents, the exact opposite of D2/D3 (the ``name_hit_bonus``
|
||||
pattern, phase 191)."""
|
||||
if v < 0:
|
||||
raise ValueError("source_name_hit_bonus must be >= 0 (0 = off)")
|
||||
return v
|
||||
|
||||
@field_validator("db_pool_size")
|
||||
@classmethod
|
||||
def _db_pool_size_positive(cls, v: int) -> int:
|
||||
|
||||
@@ -225,6 +225,13 @@ CATALOG: dict[str, SettingMeta] = {
|
||||
"name_hit_bonus": SettingMeta(
|
||||
"name_hit_bonus", "Retrieval & honesty", "Name hit bonus"
|
||||
),
|
||||
# Phase 191 (D3): the second, independent selection bonus — the SAME
|
||||
# live shape as name_hit_bonus (the selection walk reads
|
||||
# get_settings() per turn, so a saved override applies on the next
|
||||
# question).
|
||||
"source_name_hit_bonus": SettingMeta(
|
||||
"source_name_hit_bonus", "Retrieval & honesty", "Source name hit bonus"
|
||||
),
|
||||
# --- Chunking & embedding ---
|
||||
"chunk_target_chars": SettingMeta(
|
||||
"chunk_target_chars", "Chunking & embedding", "Chunk target chars"
|
||||
|
||||
@@ -39,6 +39,7 @@ from app.api.doc_drafts import router as doc_drafts_router
|
||||
from app.api.docs import router as docs_router
|
||||
from app.api.git_sources import router as git_sources_router
|
||||
from app.api.health import router as health_router
|
||||
from app.api.source_labels import router as source_labels_router
|
||||
from app.api.steering import router as steering_router
|
||||
from app.api.suggestions import router as suggestions_router
|
||||
from app.api.sync import router as sync_router
|
||||
@@ -234,6 +235,11 @@ def create_app() -> FastAPI:
|
||||
# the app_settings overrides — DB-over-env, live on Save) — admin-only
|
||||
# (router-wide require_admin; a token user stays 403).
|
||||
app.include_router(admin_settings_router, prefix="/api")
|
||||
# Phase 191 (task 02): the per-source name/label/tags surface (the
|
||||
# catalogue GET + the upsert + the reset) — admin-only (router-wide
|
||||
# require_admin; a token user stays 403). The display name is an
|
||||
# ALIAS beside the derived source name, never a rename (D1).
|
||||
app.include_router(source_labels_router, prefix="/api")
|
||||
# Phase 51: the anonymous shared-chat read — NO admin dependency.
|
||||
# /api/shared/<token> is the JSON snapshot; /shared/<token> (the
|
||||
# page route below, registered without a prefix) is the page.
|
||||
|
||||
@@ -77,6 +77,14 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
|
||||
name, the value in its env-native string form;
|
||||
the row WINS over the environment (resolved
|
||||
inside ``app.config.get_settings``).
|
||||
* ``source_labels`` — one row per source the owner has named (phase
|
||||
191, TODO.md L4): the display name ("the git
|
||||
server"), the short label/category, and the tag
|
||||
list used to focus the model when prompting.
|
||||
``source`` mirrors ``documents.source`` (the
|
||||
DERIVED name) and is an ALIAS key only — a label
|
||||
row never renames a document, a locator, or a
|
||||
citation (phase 191 D1).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -717,3 +725,57 @@ class AppSetting(Base):
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now(), nullable=False
|
||||
)
|
||||
|
||||
|
||||
class SourceLabel(Base):
|
||||
"""One owner-given name / label / tag set per source (phase 191, task 01).
|
||||
|
||||
``TODO.md`` L4: "Sources need a name, a label, tag(s) so you can
|
||||
focus the model when prompting if you have a project name or a
|
||||
standard name you reference things by." One row per SOURCE (not per
|
||||
document), keyed by the source name exactly as ``Document.source``
|
||||
carries it (the derived basename — the clone dir / local dir /
|
||||
upload file stem), so the row joins the documents and the
|
||||
``git_sources`` row without a second identity.
|
||||
|
||||
**The display name is an ALIAS ONLY (D1, owner-confirmed):**
|
||||
``Document.source``, the ``source/path`` document identity, the
|
||||
read/grep/``show_text`` locators, the citations and the KB tree are
|
||||
ALL untouched by a label row — renaming a source never invalidates
|
||||
a saved chat, a shared page or a verbatim locator. The derived name
|
||||
stays the wire identity; the display name rides beside it and is
|
||||
consumed by the retrieval boost (task 04) and the prompt surfacing
|
||||
(task 05) only.
|
||||
|
||||
No row = no name/label/tags = the pre-phase behavior exactly (the
|
||||
migration creates an EMPTY table — no backfill, A13 reversible).
|
||||
"""
|
||||
|
||||
__tablename__ = "source_labels"
|
||||
|
||||
#: Mirrors ``documents.source`` (String(120)) — the DERIVED source
|
||||
#: name, the join key and the sole PK (one label row per source;
|
||||
#: the ``folder_summaries`` PK-part precedent).
|
||||
source: Mapped[str] = mapped_column(String(120), primary_key=True)
|
||||
#: The owner's name for the source ("the git server"). Plain TEXT —
|
||||
#: the ≤ 120-char trimmed limit is enforced at the API boundary
|
||||
#: (the phase-121 ``git_sources.branch`` pattern: plain column,
|
||||
#: boundary validation). NULL = no display name (pre-phase).
|
||||
display_name: Mapped[str | None] = mapped_column(Text, default=None)
|
||||
#: The short one-line description / category ("self-hosted forge +
|
||||
#: Gitea"). Plain TEXT, ≤ 200 trimmed enforced at the API. NULL =
|
||||
#: no label.
|
||||
label: Mapped[str | None] = mapped_column(Text, default=None)
|
||||
#: The focus tokens (the owner's "tag(s)"): ≤ 10 entries, each ≤ 32
|
||||
#: chars, non-empty, de-duplicated case-insensitively — normalized
|
||||
#: and guarded at the API boundary (``app.rag.source_labels.normalize_tags``).
|
||||
#: JSONB NOT NULL with server default ``'[]'`` (the phase-89
|
||||
#: ``ignore_paths`` shape): a source with no tags reads ``[]``.
|
||||
tags: Mapped[list] = mapped_column(
|
||||
JSONB, default=list, server_default=text("'[]'"), nullable=False
|
||||
)
|
||||
#: Fresh stamp on every write (the ``sources_meta.updated_at`` house
|
||||
#: style: server-side now(), onupdate now()).
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
|
||||
)
|
||||
|
||||
+43
-4
@@ -43,6 +43,14 @@ tail (the model input stays byte-identical). The longest-prefix cap in
|
||||
one-line tail (~15 chars/source) always fit; a smaller owner-set cap
|
||||
cuts the tail first — accepted, the cap is owner-tunable.
|
||||
|
||||
Source names (phase 191, D4): when a source has a label row with a
|
||||
``display_name``, the INPUT line of its documents leads with that name
|
||||
instead of the derived source name (``build_overview_prompt``'s *names*
|
||||
map, derived from the ONE ``all_labels`` read :func:`regenerate_overview`
|
||||
performs) — the owner's vocabulary reaches the outline. The stored text,
|
||||
the catalogue rows and the deterministic ``Sources: …`` inventory tail
|
||||
keep the DERIVED name (D1 — a label is an alias, never an identity).
|
||||
|
||||
The ``KB_OVERVIEW_MODE`` marker follows the ``SUMMARY_MODE`` /
|
||||
``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on it
|
||||
in the system prompt (wired in phase 31, task 05).
|
||||
@@ -50,7 +58,7 @@ in the system prompt (wired in phase 31, task 05).
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import Sequence
|
||||
from collections.abc import Mapping, Sequence
|
||||
from datetime import UTC, datetime
|
||||
from typing import Protocol
|
||||
|
||||
@@ -62,6 +70,7 @@ from app.db import SessionLocal
|
||||
from app.models import Document, KbOverview
|
||||
from app.rag.llm import LLMError
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from app.rag.source_labels import all_labels
|
||||
|
||||
logger = logging.getLogger("app.rag.overview")
|
||||
|
||||
@@ -146,7 +155,9 @@ def _source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str:
|
||||
|
||||
|
||||
def build_overview_prompt(
|
||||
rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None
|
||||
rows: Sequence[tuple[str, str, str, str | None]],
|
||||
max_chars: int | None = None,
|
||||
names: Mapping[str, str] | None = None,
|
||||
) -> tuple[str, str]:
|
||||
"""The ``(system, user)`` message pair for one overview call.
|
||||
|
||||
@@ -166,11 +177,25 @@ def build_overview_prompt(
|
||||
cap and the shared ``[…truncated…]`` marker is appended on its own
|
||||
line, so the model never sees more than the cap and the cut is
|
||||
visible (summarizer convention, phase 30).
|
||||
|
||||
*names* (phase 191, D4) maps a DERIVED source name to the owner's
|
||||
display name for it: the input line leads with the display name
|
||||
instead of the raw source name, so the ``lite`` outline is written in
|
||||
the owner's vocabulary ("the git server" rather than
|
||||
``gitea-infra-docs``). The alias contract (D1) is untouched — the
|
||||
``path`` / ``title`` / summary halves of the line, the stored overview
|
||||
row and its ``Sources: …`` inventory tail all keep the derived name;
|
||||
only this model input changes. ``None`` / an empty map / no entry for
|
||||
a source (or an entry with an empty name) → the pre-phase line,
|
||||
byte-identical.
|
||||
"""
|
||||
limit = max_chars if max_chars is not None else get_settings().overview_input_max_chars
|
||||
lines: list[str] = []
|
||||
for source, path, title, summary in rows:
|
||||
line = f"{source} — {path} — {title}"
|
||||
# Phase 191 (D4): the display name REPLACES the raw source name in
|
||||
# the lite model's input only — never in the stored outline.
|
||||
display = names.get(source) if names else None
|
||||
line = f"{display or source} — {path} — {title}"
|
||||
first = _first_summary_line(summary)
|
||||
if first:
|
||||
line += f" — {first}"
|
||||
@@ -220,6 +245,10 @@ async def regenerate_overview(
|
||||
never sees the tail; the zero-doc and ``LLMError`` paths below
|
||||
leave a previous row byte-identical — an existing tail is part of
|
||||
the stored content and is preserved on failure.
|
||||
* Phase 191 (D4): the ONE :func:`app.rag.source_labels.all_labels`
|
||||
read happens AFTER the zero-doc gate (no label query when there is
|
||||
nothing to outline) and feeds only the model input — the stored
|
||||
content and its inventory tail stay derived-name text.
|
||||
* :class:`LLMError` → ``overview: regeneration failed — …`` is
|
||||
logged and ``False`` is returned with the previous row intact —
|
||||
an old outline is better than none.
|
||||
@@ -249,7 +278,17 @@ async def regenerate_overview(
|
||||
)
|
||||
return False
|
||||
|
||||
system, user = build_overview_prompt(rows)
|
||||
# Phase 191 (D4, task 05): ONE all_labels read for the whole run —
|
||||
# the display-name map for the model input is derived from these
|
||||
# views in this loop, never re-queried (the same single-read
|
||||
# contract the chat turn holds, task 04/05). Rows without a
|
||||
# display name are absent from the map, so their lines stay
|
||||
# byte-identical to pre-phase.
|
||||
labels = all_labels(session)
|
||||
name_map = {
|
||||
src: view.display_name for src, view in labels.items() if view.display_name
|
||||
}
|
||||
system, user = build_overview_prompt(rows, names=name_map)
|
||||
try:
|
||||
text = await llm.chat(
|
||||
[{"role": "system", "content": system}, {"role": "user", "content": user}],
|
||||
|
||||
+76
-2
@@ -100,16 +100,32 @@ owner decision). The deterministic lever is the honesty gate itself
|
||||
(A8 revised 2026-09-14: an FTS hit flips HIGH only when
|
||||
``best_cosine >= lexical_support_floor``, so the misleading docs are
|
||||
no longer injected — see :mod:`app.api.chat`).
|
||||
|
||||
Source names, labels and tags (phase 191, D4): a ``<document>`` block can
|
||||
carry the owner's label row for its source as two ADDITIVE attributes —
|
||||
``name="…"`` (the source's ``display_name``) immediately after
|
||||
``source="…"``, then ``label="…"`` (the one-line description) — so the
|
||||
model learns the name→docs mapping ("the git server" is a source with
|
||||
these documents). The attribute order is fixed
|
||||
(``source`` → ``name?`` → ``label?`` → ``path`` → ``title`` → ``date``),
|
||||
an unset field adds NOTHING, and no *names* map / no entry for the
|
||||
source leaves the block byte-identical to pre-phase (the alias contract,
|
||||
D1: the identity attributes stay the DERIVED name — a display name never
|
||||
replaces ``source`` or ``path``). Only the two new values are XML-escaped
|
||||
(they are free-form owner text); the identity attributes keep their
|
||||
pre-phase unescaped bytes.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Sequence
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
from xml.sax.saxutils import escape
|
||||
|
||||
from app.config import Settings, get_settings
|
||||
from app.models import Document
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from app.rag.source_labels import SourceLabelView
|
||||
from app.schemas import HistoryTurn
|
||||
|
||||
#: PLAN §6 verbatim (line wrapping included); ``{relevance}`` is filled by
|
||||
@@ -769,6 +785,47 @@ def _extract_context_line(image_context: str) -> str | None:
|
||||
)
|
||||
|
||||
|
||||
def _attr(value: str) -> str:
|
||||
"""XML-escape one attribute value (phase 191, D4) — ``&``/``<``/``>``
|
||||
and the double quote (``"``), the delimiter the block uses.
|
||||
|
||||
Applied ONLY to the two additive label attributes (``name`` /
|
||||
``label``): they hold free-form OWNER text where a ``"`` is realistic
|
||||
and must not break the attribute. The pre-existing identity
|
||||
attributes (``source`` / ``path`` / ``title`` / ``date``) deliberately
|
||||
keep their unescaped pre-phase bytes — escaping them here would move
|
||||
every label-less block, which the phase's byte-identity criterion
|
||||
forbids.
|
||||
"""
|
||||
return escape(value, {'"': """})
|
||||
|
||||
|
||||
def _source_label_attrs(
|
||||
doc: Document, names: Mapping[str, SourceLabelView] | None
|
||||
) -> str:
|
||||
"""The additive `` name="…"`` / `` label="…"`` fragment of one
|
||||
``<document>`` block (phase 191, D4) — the empty string unless the
|
||||
source has a label row with that field set.
|
||||
|
||||
``None`` / an empty map / no entry for ``doc.source`` / a row with
|
||||
neither field set all render ``""`` — the block is then byte-identical
|
||||
to pre-phase, and a label-only row adds exactly the one ``label``
|
||||
attribute (an unset field adds NOTHING, it is never an empty
|
||||
attribute).
|
||||
"""
|
||||
if not names:
|
||||
return ""
|
||||
view = names.get(doc.source)
|
||||
if view is None:
|
||||
return ""
|
||||
attrs = ""
|
||||
if view.display_name:
|
||||
attrs += f' name="{_attr(view.display_name)}"'
|
||||
if view.label:
|
||||
attrs += f' label="{_attr(view.label)}"'
|
||||
return attrs
|
||||
|
||||
|
||||
def _image_context_line(doc: Document) -> str | None:
|
||||
"""The phase-125 context line for one suggested document (task 05,
|
||||
LOCKED A4) — ``None`` in the common case (EVERY text doc, and every
|
||||
@@ -832,6 +889,7 @@ def build_high_prompt(
|
||||
folder_lines: Sequence[str] = (),
|
||||
show_image: bool = False,
|
||||
show_text: bool = False,
|
||||
names: Mapping[str, SourceLabelView] | None = None,
|
||||
) -> str:
|
||||
"""Grounded turn: locked persona (+ steering, + KB overview) + the
|
||||
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
|
||||
@@ -917,6 +975,21 @@ def build_high_prompt(
|
||||
(:func:`_document_body`; the NULL/whitespace-summary preview
|
||||
fallback is LOCKED A5; an enriched image doc's summary leads with
|
||||
the phase-125 one-line context line, LOCKED A4).
|
||||
|
||||
*names* (phase 191, D4): the source-name → :class:`SourceLabelView`
|
||||
map of the owner's label rows (the SAME single ``all_labels`` read
|
||||
that feeds the task-04 selection boost — one query per turn). When a
|
||||
doc's source has a row, the block head gains `` name="{display_name}"``
|
||||
immediately after ``source="…"`` and `` label="{label}"`` after that
|
||||
(the fixed order ``source`` → ``name?`` → ``label?`` → ``path`` →
|
||||
``title`` → ``date``); the two values are XML-escaped
|
||||
(:func:`_attr`), the identity attributes are not. An unset field adds
|
||||
NOTHING, so a label-only row gains exactly the one attribute, and
|
||||
``None`` / ``{}`` / no entry for the source keeps the block
|
||||
byte-identical to pre-phase (the default, and the kill pin in
|
||||
:mod:`tests.unit.test_prompts`). This is surfacing only — the alias
|
||||
contract (D1) keeps ``source`` / ``path`` as the document identity
|
||||
the tools, citations and locators use.
|
||||
"""
|
||||
# Phase 106 (D5): every document the model sees carries its
|
||||
# creation date — the block's ``date`` attribute (the row's
|
||||
@@ -926,7 +999,8 @@ def build_high_prompt(
|
||||
# stays out of the prompt (the capped ``read`` tool is the only
|
||||
# full-text path).
|
||||
blocks = [
|
||||
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
|
||||
f'<document source="{doc.source}"{_source_label_attrs(doc, names)}'
|
||||
f' path="{doc.path}" title="{doc.title}" '
|
||||
f'date="{doc.created_at:%Y-%m-%d}">\n'
|
||||
f"{_document_body(doc)}\n"
|
||||
"</document>"
|
||||
|
||||
+75
-17
@@ -291,6 +291,16 @@ class RetrievedChunk:
|
||||
:func:`fuse`'s ``replace()`` copies it (the double-hit merge ORs it
|
||||
in — a vector row that is also the name hit's representative chunk
|
||||
keeps the flag).
|
||||
* ``source_name_hit`` — True for a chunk whose parent document's
|
||||
SOURCE the question named (phase 191, D2/D3): the flag is set at
|
||||
the retrieval boundary in ``app/api/chat.py`` from the turn's ONE
|
||||
``all_labels`` read (the pure D2 rule per view — the same read that
|
||||
feeds the ``name``/``label`` prompt attributes, task 05) — the
|
||||
selection tier's SECOND bonus input (``source_name_hit_bonus``, a
|
||||
separate knob). Default
|
||||
``False`` keeps every ordinary construction valid; :func:`fuse`'s
|
||||
``replace()`` copies it (the double-hit merge ORs it in, mirroring
|
||||
``name_hit``).
|
||||
"""
|
||||
|
||||
chunk_id: uuid.UUID
|
||||
@@ -302,6 +312,7 @@ class RetrievedChunk:
|
||||
fts_hit: bool = False
|
||||
is_summary: bool = False
|
||||
name_hit: bool = False
|
||||
source_name_hit: bool = False
|
||||
|
||||
|
||||
def lexical_tsquery(question: str) -> str | None:
|
||||
@@ -339,9 +350,11 @@ def fuse(
|
||||
Lexical-only hits (no vector rank) enter with ``cosine=0.0`` and
|
||||
``fts_hit=True``; vector chunks matched by the lexical list get
|
||||
``fts_hit=True`` in place (the input objects are mutated — callers
|
||||
should not reuse them afterwards), and ``name_hit=True`` is ORed in
|
||||
should not reuse them afterwards), ``name_hit=True`` is ORed in
|
||||
(a vector row that is also the name hit's representative chunk is a
|
||||
name-hit row — the phase-119 selection tier must see it).
|
||||
name-hit row — the phase-119 selection tier must see it), and
|
||||
``source_name_hit`` is ORed in the SAME way (phase 191, D3 — a
|
||||
vector row merged with a flagged source-name row keeps the flag).
|
||||
"""
|
||||
if k <= 0:
|
||||
raise ValueError("rrf k must be > 0")
|
||||
@@ -355,7 +368,10 @@ def fuse(
|
||||
if rc.chunk_id in by_id:
|
||||
existing = by_id[rc.chunk_id]
|
||||
by_id[rc.chunk_id] = replace(
|
||||
existing, fts_hit=True, name_hit=existing.name_hit or rc.name_hit
|
||||
existing,
|
||||
fts_hit=True,
|
||||
name_hit=existing.name_hit or rc.name_hit,
|
||||
source_name_hit=existing.source_name_hit or rc.source_name_hit,
|
||||
)
|
||||
fused[rc.chunk_id] += term
|
||||
else:
|
||||
@@ -754,17 +770,23 @@ def retrieve(
|
||||
def _selection_order(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
bonus: float,
|
||||
source_bonus: float = 0.0,
|
||||
) -> list[tuple[Document, float, float, int]]:
|
||||
"""The shared selection walk of the phase-119 name-hit bonus (D2,
|
||||
LOCKED A3) — one walk for ``select_suggested``, ``select_related``,
|
||||
and ``weak_hit_titles``.
|
||||
and ``weak_hit_titles``, carrying the phase-191 source-name bonus
|
||||
(D3) as its SECOND, independent knob.
|
||||
|
||||
Returns ``(document, effective_score, best_cosine, first_seen_index)``
|
||||
for each distinct document, where *effective_score* is the
|
||||
document's best fused chunk score plus *bonus* when ANY of its
|
||||
chunks carries ``name_hit`` (the D1 path match — the bonus is per
|
||||
DOCUMENT, applied ONCE no matter how many of the document's chunks
|
||||
are name hits).
|
||||
are name hits), plus *source_bonus* when ANY of its chunks carries
|
||||
``source_name_hit`` (the phase-191 D2 source-name match — the SAME
|
||||
per-DOCUMENT, apply-ONCE rule; at most TWO bonuses per document
|
||||
on the fused scale, bounded: name_hit_bonus + source_name_hit_bonus
|
||||
= 0.010 at the shipped defaults, never more).
|
||||
|
||||
The walk keeps the EXISTING selection semantics (not just the loop):
|
||||
the same stable score-descending order as
|
||||
@@ -773,16 +795,18 @@ def _selection_order(
|
||||
best cosine tracked across ALL of its chunks (the tiered walk's
|
||||
tracking).
|
||||
|
||||
* *bonus* ``== 0`` (the kill switch) or no name-hit chunk present:
|
||||
* *bonus* ``== 0`` (the kill switch) or no name-hit chunk present,
|
||||
AND *source_bonus* ``== 0`` or no source-name-hit chunk present:
|
||||
the document order is IDENTICAL to the pre-phase walk — no re-sort
|
||||
happens at all (byte-identical, LOCKED A3);
|
||||
happens at all (byte-identical, LOCKED A3, mirrored for the second
|
||||
bonus by LOCKED A3 of phase 191);
|
||||
* otherwise the documents are ordered by
|
||||
``(−effective_score, −best_cosine, document.path,
|
||||
first_seen_index)`` — the bounded re-rank: a name-hit document
|
||||
gets a head start on the fused scale, and an effective-score tie
|
||||
resolves by cosine, then path, then the pre-bonus rank.
|
||||
|
||||
The bonus lives in the SELECTION layer only: the chunk objects are
|
||||
The bonuses live in the SELECTION layer only: the chunk objects are
|
||||
never modified — their ``score``/``cosine``/``fts_hit`` (the A8
|
||||
gate's inputs) and ``query_log.top_score`` are untouched.
|
||||
"""
|
||||
@@ -790,6 +814,7 @@ def _selection_order(
|
||||
best_score: dict[uuid.UUID, float] = {}
|
||||
best_cosine: dict[uuid.UUID, float] = {}
|
||||
any_name_hit: dict[uuid.UUID, bool] = {}
|
||||
any_source_hit: dict[uuid.UUID, bool] = {}
|
||||
first_seen: dict[uuid.UUID, int] = {}
|
||||
for idx, rc in enumerate(sorted(chunks, key=lambda c: c.score, reverse=True)):
|
||||
doc = rc.document
|
||||
@@ -798,17 +823,24 @@ def _selection_order(
|
||||
best_cosine[doc.id] = rc.cosine
|
||||
if rc.name_hit:
|
||||
any_name_hit[doc.id] = True
|
||||
if rc.source_name_hit:
|
||||
any_source_hit[doc.id] = True
|
||||
else:
|
||||
order.append(doc)
|
||||
best_score[doc.id] = rc.score
|
||||
best_cosine[doc.id] = rc.cosine
|
||||
any_name_hit[doc.id] = rc.name_hit
|
||||
any_source_hit[doc.id] = rc.source_name_hit
|
||||
first_seen[doc.id] = idx
|
||||
effective = {
|
||||
doc.id: best_score[doc.id] + (bonus if any_name_hit[doc.id] else 0.0)
|
||||
doc.id: best_score[doc.id]
|
||||
+ (bonus if any_name_hit[doc.id] else 0.0)
|
||||
+ (source_bonus if any_source_hit[doc.id] else 0.0)
|
||||
for doc in order
|
||||
}
|
||||
if bonus > 0.0 and any(any_name_hit[doc.id] for doc in order):
|
||||
if (bonus > 0.0 and any(any_name_hit[doc.id] for doc in order)) or (
|
||||
source_bonus > 0.0 and any(any_source_hit[doc.id] for doc in order)
|
||||
):
|
||||
order.sort(
|
||||
key=lambda doc: (
|
||||
-effective[doc.id],
|
||||
@@ -826,17 +858,26 @@ def _selection_order(
|
||||
def weak_hit_titles(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
bonus: float | None = None,
|
||||
source_bonus: float | None = None,
|
||||
) -> list[str]:
|
||||
"""Distinct parent-document titles of *chunks*, best SELECTION score
|
||||
first (the phase-119 name-hit bonus applied — see
|
||||
:func:`_selection_order`, D2, LOCKED A3).
|
||||
first (the phase-119 name-hit bonus and the phase-191 source-name
|
||||
bonus applied — see :func:`_selection_order`, D2/D3, LOCKED A3).
|
||||
|
||||
Deflection mode (PLAN §6, A8) is built from these *titles only* — the
|
||||
LOW prompt and the "Maybe try" chips never see document content.
|
||||
|
||||
*source_bonus* defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting
|
||||
(0.005); ``0`` reproduces the pre-phase-191 walk byte-identically.
|
||||
"""
|
||||
if bonus is None:
|
||||
bonus = get_settings().name_hit_bonus
|
||||
return [doc.title for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)]
|
||||
if source_bonus is None:
|
||||
source_bonus = get_settings().source_name_hit_bonus
|
||||
return [
|
||||
doc.title
|
||||
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus)
|
||||
]
|
||||
|
||||
|
||||
def select_documents_tiered(
|
||||
@@ -953,6 +994,7 @@ def select_suggested(
|
||||
chunks: Sequence[RetrievedChunk],
|
||||
n: int | None = None,
|
||||
bonus: float | None = None,
|
||||
source_bonus: float | None = None,
|
||||
) -> list[Document]:
|
||||
"""Top-N distinct parent documents in SELECTION rank order — the
|
||||
phase-118 "start here" suggestion tier (LOCKED A3, re-revised by
|
||||
@@ -975,7 +1017,10 @@ def select_suggested(
|
||||
|
||||
*bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the
|
||||
owner-tunable starting point); ``0`` reproduces the pre-phase walk
|
||||
byte-identically (the kill switch, LOCKED A3).
|
||||
byte-identically (the kill switch, LOCKED A3). *source_bonus*
|
||||
defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting (phase 191,
|
||||
D3 — the same bounded shape, the same kill switch, an independent
|
||||
knob).
|
||||
|
||||
The returned rows carry the full document content, byte-identical —
|
||||
the content is what the agent's ``read`` tool serves later (never
|
||||
@@ -985,7 +1030,14 @@ def select_suggested(
|
||||
top_n = n if n is not None else get_settings().suggested_docs
|
||||
if bonus is None:
|
||||
bonus = get_settings().name_hit_bonus
|
||||
return [doc for doc, _eff, _cos, _idx in _selection_order(chunks, bonus)[:top_n]]
|
||||
if source_bonus is None:
|
||||
source_bonus = get_settings().source_name_hit_bonus
|
||||
return [
|
||||
doc
|
||||
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus)[
|
||||
:top_n
|
||||
]
|
||||
]
|
||||
|
||||
|
||||
def select_related(
|
||||
@@ -993,6 +1045,7 @@ def select_related(
|
||||
excluded_ids: set[uuid.UUID],
|
||||
cap: int,
|
||||
bonus: float | None = None,
|
||||
source_bonus: float | None = None,
|
||||
) -> list[Document]:
|
||||
"""The documents ranked AFTER *excluded_ids* — the phase-118 related
|
||||
tier (rank 6+ for the contiguous top-5 suggestion set), up to *cap*
|
||||
@@ -1012,7 +1065,10 @@ def select_related(
|
||||
|
||||
*bonus* defaults to the ``BOR_NAME_HIT_BONUS`` setting (0.005 — the
|
||||
owner-tunable starting point); ``0`` reproduces the pre-phase walk
|
||||
byte-identically (the kill switch, LOCKED A3).
|
||||
byte-identically (the kill switch, LOCKED A3). *source_bonus*
|
||||
defaults to the ``BOR_SOURCE_NAME_HIT_BONUS`` setting (phase 191,
|
||||
D3 — the same bounded shape, the same kill switch, an independent
|
||||
knob).
|
||||
|
||||
The returned rows carry the full document content, byte-identical
|
||||
(the tier is metadata for the ``done`` frame and the durable
|
||||
@@ -1020,8 +1076,10 @@ def select_related(
|
||||
"""
|
||||
if bonus is None:
|
||||
bonus = get_settings().name_hit_bonus
|
||||
if source_bonus is None:
|
||||
source_bonus = get_settings().source_name_hit_bonus
|
||||
out: list[Document] = []
|
||||
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus):
|
||||
for doc, _eff, _cos, _idx in _selection_order(chunks, bonus, source_bonus):
|
||||
if len(out) >= cap:
|
||||
break
|
||||
if doc.id in excluded_ids:
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
"""Source name/label/tags storage + the D2 source-name matching rule (phase 191, task 01).
|
||||
|
||||
``TODO.md`` L4: "Sources need a name, a label, tag(s) so you can focus
|
||||
the model when prompting if you have a project name or a standard name
|
||||
you reference things by." This module is the whole non-HTTP surface of
|
||||
that feature: the upsert/get/delete helpers over the ``source_labels``
|
||||
table (migration 0030) and the pure matching rule the retrieval boost
|
||||
(task 04) consumes.
|
||||
|
||||
**The alias contract (D1, owner-confirmed):** a label row is an ALIAS
|
||||
beside the derived source name — ``Document.source``, the ``source/path``
|
||||
document identity, the read/grep/``show_text`` locators, the citations
|
||||
and the KB tree are ALL untouched by it. Renaming a source never
|
||||
invalidates a saved chat, a shared page or a verbatim locator. Every
|
||||
function here keys on the DERIVED name (``Document.source``); the
|
||||
display name is data, never identity.
|
||||
|
||||
**The matching rule (D2):** a source is a SOURCE-NAME HIT when at least
|
||||
one :func:`app.rag.retriever.name_hit_tokens` candidate (the phase-119
|
||||
class-agnostic token list, reused UNCHANGED) equals — in the
|
||||
:func:`app.rag.retriever._normalize_name` form, so case-insensitive and
|
||||
accent-folded (phase 162) — a token of the source's ``display_name``,
|
||||
``label`` or any ``tags`` entry. The phase-119 digit-class rule does
|
||||
NOT apply to user-given names (the owner's names are not paths —
|
||||
"gitea" must hit even though it is digitless; the digit rule exists to
|
||||
keep prose tokens from matching PATHS). The one filter is
|
||||
:data:`app.rag.retriever.NAME_TOKEN_MIN_LEN` (4) applied to the SOURCE
|
||||
side too — a 2-char tag never matches.
|
||||
|
||||
Session convention (the phase-53 ``sources_meta`` style): the write
|
||||
helpers only *flush* — the **caller** commits.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from typing import Final
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.models import SourceLabel
|
||||
from app.rag.retriever import NAME_TOKEN_MIN_LEN, _normalize_name, name_hit_tokens
|
||||
|
||||
#: Boundary limits (phase 191 D1/D5) — the API (task 02) turns the
|
||||
#: ``normalize_tags`` raises into the framework's standard 422; the
|
||||
#: display-name / label caps are enforced there against these constants.
|
||||
DISPLAY_NAME_MAX_CHARS: Final = 120
|
||||
LABEL_MAX_CHARS: Final = 200
|
||||
TAG_MAX_CHARS: Final = 32
|
||||
TAGS_MAX_COUNT: Final = 10
|
||||
|
||||
|
||||
class _Unset:
|
||||
"""Marker type for "field omitted" in :func:`set_label`.
|
||||
|
||||
The partial-update shape (phase-168 D5): an OMITTED field leaves the
|
||||
stored value untouched, an explicit ``None`` CLEARS it — the two are
|
||||
indistinguishable with plain ``None`` defaults, so the sentinel owns
|
||||
the difference (the API passes only the fields the request carried).
|
||||
"""
|
||||
|
||||
__slots__ = ()
|
||||
|
||||
|
||||
_UNSET: Final = _Unset()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceLabelView:
|
||||
"""Immutable read-side snapshot of one label row (D1).
|
||||
|
||||
``tags`` is a ``tuple`` so the view stays hashable/mutable-proof
|
||||
after the ORM row leaves the session; ``display_name`` / ``label``
|
||||
are ``None`` when unset (never the empty string — the API trims
|
||||
empty submissions to NULL).
|
||||
"""
|
||||
|
||||
source: str
|
||||
display_name: str | None
|
||||
label: str | None
|
||||
tags: tuple[str, ...]
|
||||
|
||||
|
||||
def _view(row: SourceLabel) -> SourceLabelView:
|
||||
return SourceLabelView(
|
||||
source=row.source,
|
||||
display_name=row.display_name,
|
||||
label=row.label,
|
||||
tags=tuple(row.tags or ()),
|
||||
)
|
||||
|
||||
|
||||
def get_label(db: Session, source: str) -> SourceLabel | None:
|
||||
"""The ORM row for *source* (one PK read); ``None`` when absent."""
|
||||
return db.get(SourceLabel, source)
|
||||
|
||||
|
||||
def label_for(db: Session, source: str) -> SourceLabelView | None:
|
||||
"""The frozen :class:`SourceLabelView` for *source* (one PK read);
|
||||
``None`` when absent — the "no label row = pre-phase" contract."""
|
||||
row = db.get(SourceLabel, source)
|
||||
return None if row is None else _view(row)
|
||||
|
||||
|
||||
def all_labels(db: Session) -> dict[str, SourceLabelView]:
|
||||
"""Every label row as a frozen view, keyed by ``source``.
|
||||
|
||||
ONE query (the phase-119 one-query convention): the table carries a
|
||||
handful of rows, so callers match in Python against it instead of
|
||||
issuing a query per source. Ordered by ``source`` for determinism.
|
||||
"""
|
||||
rows = db.scalars(select(SourceLabel).order_by(SourceLabel.source)).all()
|
||||
return {row.source: _view(row) for row in rows}
|
||||
|
||||
|
||||
def set_label(
|
||||
db: Session,
|
||||
source: str,
|
||||
*,
|
||||
display_name: str | None | _Unset = _UNSET,
|
||||
label: str | None | _Unset = _UNSET,
|
||||
tags: Sequence[str] | None | _Unset = _UNSET,
|
||||
) -> SourceLabel:
|
||||
"""Upsert the label row for *source*; return the ORM row.
|
||||
|
||||
The partial-update shape (phase 191 D5, the phase-168 precedent):
|
||||
a field left at the ``_UNSET`` sentinel keeps its stored value, an
|
||||
explicitly passed value OVERWRITES it, an explicit ``None`` CLEARS
|
||||
(``tags`` clears to ``[]`` — the column is NOT NULL). A missing row
|
||||
is created with the passed values (omitted fields = their defaults:
|
||||
``NULL`` / ``[]``). Values arrive ALREADY VALIDATED (the API layer
|
||||
trims/enforces the caps via :func:`normalize_tags`); this helper
|
||||
stores what it is given.
|
||||
|
||||
Flush-only (the phase-53 session convention): the **caller** commits.
|
||||
"""
|
||||
row = db.get(SourceLabel, source)
|
||||
if row is None:
|
||||
row = SourceLabel(
|
||||
source=source,
|
||||
display_name=None if isinstance(display_name, _Unset) else display_name,
|
||||
label=None if isinstance(label, _Unset) else label,
|
||||
tags=[] if isinstance(tags, _Unset) or tags is None else list(tags),
|
||||
)
|
||||
db.add(row)
|
||||
else:
|
||||
if not isinstance(display_name, _Unset):
|
||||
row.display_name = display_name
|
||||
if not isinstance(label, _Unset):
|
||||
row.label = label
|
||||
if not isinstance(tags, _Unset):
|
||||
row.tags = [] if tags is None else list(tags)
|
||||
db.flush()
|
||||
return row
|
||||
|
||||
|
||||
def delete_label(db: Session, source: str) -> bool:
|
||||
"""Remove the label row for *source*; ``False`` when absent (the
|
||||
API's 404 shape), ``True`` when a row was deleted. Flush-only."""
|
||||
row = db.get(SourceLabel, source)
|
||||
if row is None:
|
||||
return False
|
||||
db.delete(row)
|
||||
db.flush()
|
||||
return True
|
||||
|
||||
|
||||
def normalize_tags(raw: Sequence[str]) -> list[str]:
|
||||
"""Normalize one tag submission (pure, the API's 422 source).
|
||||
|
||||
* trim every entry, drop the ones that are empty afterwards
|
||||
(``[]`` is legal — it clears the tags);
|
||||
* raise :class:`ValueError` above :data:`TAGS_MAX_COUNT` entries
|
||||
(counted on the trimmed non-empty list, BEFORE the case-insensitive
|
||||
de-dupe: 11 typed tags are 11 rejected tags);
|
||||
* raise :class:`ValueError` when any entry exceeds
|
||||
:data:`TAG_MAX_CHARS` afterwards;
|
||||
* de-duplicate case-insensitively, preserving the FIRST case seen
|
||||
(``Gitea`` then ``gitea`` keeps ``Gitea``), order otherwise kept.
|
||||
"""
|
||||
cleaned = [tag.strip() for tag in raw]
|
||||
cleaned = [tag for tag in cleaned if tag]
|
||||
if len(cleaned) > TAGS_MAX_COUNT:
|
||||
raise ValueError(f"at most {TAGS_MAX_COUNT} tags are allowed")
|
||||
for tag in cleaned:
|
||||
if len(tag) > TAG_MAX_CHARS:
|
||||
raise ValueError(f"each tag must be at most {TAG_MAX_CHARS} characters")
|
||||
out: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for tag in cleaned:
|
||||
key = tag.casefold()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
out.append(tag)
|
||||
return out
|
||||
|
||||
|
||||
def _source_side_tokens(view: SourceLabelView) -> set[str]:
|
||||
"""The normalized token set of one label row (the D2 source side).
|
||||
|
||||
Every whitespace token of ``display_name`` / ``label`` / every
|
||||
``tags`` entry, folded through :func:`_normalize_name` (the single
|
||||
normalization BOTH sides of name matching pass through, phase 162)
|
||||
and kept at :data:`NAME_TOKEN_MIN_LEN` chars minimum — the 4-char
|
||||
floor applies to the SOURCE side too (a 2-char tag never matches).
|
||||
"""
|
||||
tokens: set[str] = set()
|
||||
fields: list[str] = [view.display_name or "", view.label or "", *view.tags]
|
||||
for field in fields:
|
||||
for raw in field.split():
|
||||
norm = _normalize_name(raw)
|
||||
if len(norm) >= NAME_TOKEN_MIN_LEN:
|
||||
tokens.add(norm)
|
||||
return tokens
|
||||
|
||||
|
||||
def question_name_hits(view: SourceLabelView | None, question: str) -> bool:
|
||||
"""The D2 rule (pure): does *question* name the source behind *view*?
|
||||
|
||||
``None`` (no label row) → ``False`` — the pre-phase contract. Any
|
||||
:func:`name_hit_tokens` candidate EQUAL (in the normalized form) to
|
||||
a source-side token is a hit: no digit-class rule, no prefix match
|
||||
(user-given names have no prose-precision problem — the phase-119
|
||||
digit rule guards PATHS, not names).
|
||||
"""
|
||||
if view is None:
|
||||
return False
|
||||
candidates = name_hit_tokens(question)
|
||||
if not candidates:
|
||||
return False
|
||||
source_tokens = _source_side_tokens(view)
|
||||
return any(candidate in source_tokens for candidate in candidates)
|
||||
|
||||
|
||||
def question_name_hit_sources(db: Session, question: str) -> set[str]:
|
||||
"""The derived source names the question names (D2) — the standalone
|
||||
one-query entry point (no other per-turn work needed).
|
||||
|
||||
ONE :func:`all_labels` read + the pure rule per row (a handful of
|
||||
rows, matched in Python — the phase-119 one-query convention). The
|
||||
empty set is the common case: no label rows, or no name in the
|
||||
question.
|
||||
|
||||
The chat turn does NOT call this: since task 05 it inlines the same
|
||||
pair against its own single ``all_labels`` read — the read that also
|
||||
feeds the prompt surfaces (D4) — so a turn issues ONE label query.
|
||||
"""
|
||||
return {
|
||||
view.source
|
||||
for view in all_labels(db).values()
|
||||
if question_name_hits(view, question)
|
||||
}
|
||||
+115
@@ -2246,3 +2246,118 @@ class AdminSettingsCleared(BaseModel):
|
||||
deleted."""
|
||||
|
||||
cleared: int
|
||||
|
||||
|
||||
#: One focus tag (phase 191, D1/D5): the owner's short token, capped at
|
||||
#: the ``app.rag.source_labels.TAG_MAX_CHARS`` 32 (the ``_Chip`` item-cap
|
||||
#: precedent — an ``Annotated`` alias, so the JSON shape stays a plain
|
||||
#: string and the cap rides on every list of tags, request AND response).
|
||||
_Tag = Annotated[str, Field(min_length=1, max_length=32)]
|
||||
|
||||
|
||||
class SourceLabelOut(BaseModel):
|
||||
"""One source's name/label/tags (``GET /api/source-labels`` entry,
|
||||
``PUT``/``DELETE`` response — phase 191, task 02).
|
||||
|
||||
``source`` is the DERIVED source name — the name ``Document.source``
|
||||
carries (the clone dir / local dir / upload file stem): the row's
|
||||
identity on the wire and the join key to the documents
|
||||
(phase 191 D1). The display name is an ALIAS beside it: no locator,
|
||||
citation or KB-tree node is renamed by a label row.
|
||||
|
||||
``display_name`` / ``label`` are ``null`` when the owner set nothing
|
||||
(never the empty string — the request validator folds a
|
||||
whitespace-only submission to ``None``), ``tags`` ``[]``. No row =
|
||||
the same null/``[]`` shape: a source without a label row is
|
||||
indistinguishable from one whose label was cleared, which is exactly
|
||||
the pre-phase state (D1).
|
||||
|
||||
``extra="forbid"`` (the :class:`GitSourceOut` structural contract):
|
||||
a key outside this shape cannot be built into a response — the
|
||||
row's ``updated_at`` and anything else that leaks later raise at
|
||||
construction instead of silently widening the wire shape.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
#: The derived source name (``documents.source``, ``String(120)``).
|
||||
source: str = Field(max_length=120)
|
||||
display_name: str | None = Field(default=None, max_length=120)
|
||||
label: str | None = Field(default=None, max_length=200)
|
||||
#: At most 10 tags of at most 32 chars each (:data:`_Tag`).
|
||||
tags: list[_Tag] = Field(default_factory=list, max_length=10)
|
||||
|
||||
|
||||
class SourceLabelList(BaseModel):
|
||||
"""``GET /api/source-labels`` response (phase 191, task 02): the
|
||||
whole source catalogue — every ``git_sources`` row (git AND local,
|
||||
or the ``BOR_GIT_SOURCES`` env names while the table is empty) plus
|
||||
every indexed document source no row derives (the uploads), one
|
||||
entry per name, de-duplicated and sorted by ``source`` (the stable
|
||||
order the Sources page and the phase's E2E rely on). Each entry
|
||||
carries its label row or the null/``[]`` defaults."""
|
||||
|
||||
labels: list[SourceLabelOut]
|
||||
|
||||
|
||||
class SourceLabelUpdate(BaseModel):
|
||||
"""``PUT /api/source-labels/{source}`` body (phase 191, task 02).
|
||||
|
||||
The partial-update shape (D5, the phase-168 precedent): an OMITTED
|
||||
field leaves the stored value untouched, an explicit ``null`` CLEARS
|
||||
it, a value SETS it — the two ``None`` legs are distinguishable
|
||||
because the router reads ``model_fields_set``, not the value.
|
||||
|
||||
Every string is trimmed *before* the length constraints run (the
|
||||
``GitSourceIn._trim_url`` precedent), so a value with surrounding
|
||||
spaces stores clean and a whitespace-only ``display_name``/``label``
|
||||
folds to ``None`` — the CLEAR leg, since storing an empty string
|
||||
would make "no name" and "an empty name" two different things the
|
||||
prompt builders (task 05) would have to distinguish.
|
||||
|
||||
``tags`` carries the RAW list the token input produced. Every entry
|
||||
is trimmed by ``_trim_tags`` BEFORE the constraints run, so an entry
|
||||
that is empty afterwards is the framework's standard 422 (an empty
|
||||
focus token matches nothing — a silent drop would tell the owner
|
||||
"saved" while their tag vanished); the count cap (10) and the entry
|
||||
cap (32, :data:`_Tag`) are the same numbers
|
||||
:func:`app.rag.source_labels.normalize_tags` guards, and the router
|
||||
still runs the submission through it for the case-insensitive
|
||||
de-dupe (first case wins) and maps its ``ValueError`` to a 422, so
|
||||
the rule can never turn into a 500 if this shape is ever relaxed.
|
||||
``[]`` is legal and clears.
|
||||
|
||||
``extra="forbid"`` rejects a key outside the shape (the phase-91
|
||||
voice) — a client cannot half-write a field nobody declared.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
display_name: str | None = Field(default=None, max_length=120)
|
||||
label: str | None = Field(default=None, max_length=200)
|
||||
tags: list[_Tag] | None = Field(default=None, max_length=10)
|
||||
|
||||
@field_validator("display_name", "label", mode="before")
|
||||
@classmethod
|
||||
def _trim_to_none(cls, v: object) -> object:
|
||||
"""Trim, and fold an empty result to ``None`` (the CLEAR leg).
|
||||
|
||||
``mode='before'`` so the trim runs BEFORE ``max_length`` (the
|
||||
``GitSourceIn._trim_url`` convention): 121 chars that trim to
|
||||
120 are legal, 121 chars of text are not."""
|
||||
if isinstance(v, str):
|
||||
trimmed = v.strip()
|
||||
return trimmed or None
|
||||
return v
|
||||
|
||||
@field_validator("tags", mode="before")
|
||||
@classmethod
|
||||
def _trim_tags(cls, v: object) -> object:
|
||||
"""Trim every entry BEFORE the item constraints run (the same
|
||||
``GitSourceIn`` ordering as :meth:`_trim_to_none`): ``" gitea "``
|
||||
passes as ``gitea``, ``" "`` becomes ``""`` and the item's
|
||||
``min_length`` rejects it. Non-list / non-string input is passed
|
||||
through untouched so pydantic reports it in its own voice."""
|
||||
if isinstance(v, list):
|
||||
return [tag.strip() if isinstance(tag, str) else tag for tag in v]
|
||||
return v
|
||||
|
||||
Reference in New Issue
Block a user