phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
2026-09-16 06:57:49 -04:00
parent 21aad84a6d
commit 9820c361b0
80 changed files with 4690 additions and 1302 deletions
+104 -57
View File
@@ -37,13 +37,30 @@ Steering (phase 15): the owner's stored tuning notes are loaded per turn
section — both the HIGH and the LOW prompt carry it. The per-turn log
line records ``tuning=N`` (the number of injected notes).
Summaries (phase 30): a lite-model summary chunk's parent *is* the
source document, so a summary hit resolves to the full source document
through the unchanged chunk→document mapping (A7 revised) — context
assembly is untouched. ``TurnPlan.summary_hits`` counts the hit chunks
with ``is_summary`` whose parent document landed in the selected
top-N context, and the per-turn log line records ``summary_hits=N``
after ``fts_hits`` (PLAN §9 line extension).
Summaries (phase 30; phase 118 redefinition): a lite-model summary
chunk's parent *is* the source document, so a summary hit resolves to
the full source document through the unchanged chunk→document mapping
(A7 revised). ``TurnPlan.summary_hits`` counts the hit chunks with
``is_summary`` whose parent document is in the SUGGESTED set (phase 118:
redefined from the phase-113 cited set — the suggestion tier is the
seeded context now), and the per-turn log line records ``summary_hits=N``
after ``fts_hits`` (PLAN §9 line extension; phase 118 adds
``suggested=N`` after ``summary_hits=N``).
Summary seed context (phase 118, TODO L3 — the owner directive that
re-revises A7, LOCKED A6): a grounded turn's ``<documents>`` section
seeds the top-5 suggested documents' SUMMARIES (never their full
texts) — the "start here if these summaries seem right to you" starting
point — and the LLM extends its context by ``read``-ing what it needs
(the capped ``read`` tool is the ONLY full-text path). Both tiers are
computed once per turn, for BOTH branches: ``suggested_docs`` (no
floor, A3) and ``related_docs`` (rank 6+, the done frame's row).
``done.sources`` is the citation surface — suggested + agent-read,
deduped (A4: a grounded turn always shows chips); ``query_log.sources``
and the per-turn log line keep recording the full retrieval
(suggested + related + read, LOCKED A3). The deflected branch's prompt
is byte-identical (weak-hit titles only — A8 untouched); its TurnPlan
still carries both tiers for the durable record.
KB overview (phase 31): the lite-generated outline of the knowledge
base (single ``kb_overview`` row) is read per turn (one indexed PK
@@ -195,7 +212,13 @@ from app.rag.llm import (
)
from app.rag.overview import load_kb_overview
from app.rag.prompts import build_deflect_prompt, build_high_prompt, history_to_messages
from app.rag.retriever import RetrievedChunk, retrieve, select_documents_tiered, weak_hit_titles
from app.rag.retriever import (
RetrievedChunk,
retrieve,
select_related,
select_suggested,
weak_hit_titles,
)
from app.rag.scaffolding import ScaffoldingFilter # phase 71: the streaming filter
from app.rag.suggestions import derive_suggestions
from app.schemas import (
@@ -258,12 +281,19 @@ class TurnPlan:
fts_hits: int # lexical (OR-tsquery) candidates matched
deflected: bool
system_prompt: str
docs: list[Document] # cited sources (phase 113: the bar-clearing tier)
related_docs: list[Document] = field(default_factory=list) # phase 113
# The summary-seeded suggestion tier (phase 118, A6: top-N distinct
# documents, NO floor, A3 — the HIGH prompt carries their SUMMARIES;
# the done frame's citation surface + run_agent's seed_docs, A4).
suggested_docs: list[Document]
# Rank 6+ after the suggested set (phase 118: the next ranked docs
# that are not already suggested, at most related_max_docs — the
# done frame's de-emphasized "nearby docs" row, never a citation).
related_docs: list[Document] = field(default_factory=list)
suggestions: list[str] = field(default_factory=list) # "Maybe try" chips (deflected turns only)
tuning_count: int = 0 # steering notes injected into the system prompt
#: Hit chunks with ``is_summary`` whose parent document made it into
#: *docs* (phase 30; per-turn log line ``summary_hits=N``).
#: Hit chunks with ``is_summary`` whose parent document is in the
#: SUGGESTED set (phase 30; redefined from the phase-113 cited set
#: in phase 118; per-turn log line ``summary_hits=N``).
summary_hits: int = 0
#: Length of the stored KB overview injected as the
#: ``<knowledge_base>`` section (phase 31; per-turn log line
@@ -281,7 +311,8 @@ def plan_turn(
* **HIGH (grounded)** when ``best_cosine >= threshold`` **or**
(``fts_hits > 0`` **and** ``best_cosine >= lexical_support_floor``):
HIGH prompt with the full top-N documents, no suggestions. A cosine
HIGH prompt seeded with the suggested documents' SUMMARIES (phase
118, A6 — never their full texts), no suggestions. A cosine
exactly at the threshold is an answer — the gate is strict
(``< threshold``). An FTS hit alone, without vector corroboration
(cosine < lexical_support_floor), stays LOW (A8 revised 2026-09-14).
@@ -305,42 +336,52 @@ def plan_turn(
prompts are byte-identical to the pre-phase text and ``kb_chars``
is 0.
``summary_hits`` (phase 30) counts the hit chunks with
``is_summary`` whose parent document is among the selected
top-N documents — both the HIGH and the LOW branch record it.
``summary_hits`` (phase 30; phase 118 redefinition) counts the hit
chunks with ``is_summary`` whose parent document is in the
SUGGESTED set — both the HIGH and the LOW branch record it.
Phase 113 (the usefulness bar, LOCKED A2): retrieval documents are
tiered before either branch — ``docs`` (cited) are the distinct
parent documents whose best hit-chunk cosine clears
``settings.source_usefulness_floor`` (at most ``top_n_docs`` — the
ceiling, never a quota); ``related_docs`` are the next ranked
documents (at most ``related_max_docs``) that did not earn a cited
slot. On a deflected turn the weak hits fall to ``related_docs``
(the cited tier is usually empty — nothing below the bar earned a
citation slot); the LOW prompt itself is unchanged (weak-hit titles
only).
Phase 118 (the summary seeding, LOCKED A3/A6): retrieval documents
are tiered once, before either branch — ``suggested_docs`` are the
top-N distinct parent documents in fused rank order with NO cosine
floor (``settings.suggested_docs``, default 5 — the "start here"
seeding: the HIGH prompt carries their SUMMARIES, never their full
texts; the floor never filters, so a lexical-only hit is suggested
when it ranks) and ``related_docs`` are the next ranked documents
that are not already suggested (rank 6+ for the contiguous top-5
suggestion set, at most ``related_max_docs`` — the done frame's
de-emphasized "nearby docs" row). On a deflected turn the weak hits
are suggested too (no floor); the LOW prompt itself is unchanged
(weak-hit titles only), and both tiers still ride the TurnPlan for
the durable record (LOCKED A3: query_log records retrieval, not
citations).
``settings.top_n_docs`` and ``settings.source_usefulness_floor`` are
NOT consulted here (phase 118 retired their seeding role, A6) —
they remain settings for env back-compat only.
"""
steering = list(notes or [])
kb_text = (kb_overview or "").strip()
kb_chars = len(kb_text)
best_cosine = max((c.cosine for c in chunks), default=0.0)
fts_hits = sum(1 for c in chunks if c.fts_hit)
docs, related_docs = select_documents_tiered(
chunks,
n=settings.top_n_docs,
floor=settings.source_usefulness_floor,
related_cap=settings.related_max_docs,
# Phase 118 (A3/A6): BOTH tiers, computed once, for BOTH branches —
# the suggested tier (top-N, NO floor) seeds the HIGH prompt and the
# agent; the related tier (rank 6+ after the suggested set) feeds
# the done frame's row and the durable record.
suggested = select_suggested(chunks, n=settings.suggested_docs)
related_docs = select_related(
chunks, {d.id for d in suggested}, settings.related_max_docs
)
selected_ids = {d.id for d in docs}
summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in selected_ids)
suggested_ids = {d.id for d in suggested}
summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in suggested_ids)
lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor
if best_cosine >= settings.relevance_threshold or lexical_supported:
return TurnPlan(
best_cosine,
fts_hits,
False,
build_high_prompt(docs, notes=steering, kb_overview=kb_text),
docs,
build_high_prompt(suggested, notes=steering, kb_overview=kb_text),
suggested,
related_docs,
[],
len(steering),
@@ -353,7 +394,7 @@ def plan_turn(
fts_hits,
True,
build_deflect_prompt(titles, notes=steering, kb_overview=kb_text),
docs,
suggested,
related_docs,
derive_suggestions(titles, settings.suggestions),
len(steering),
@@ -606,7 +647,7 @@ async def chat(
db_factory, # SEC-14-04: session factory, not a long-lived session
system_prompt=plan.system_prompt,
user_message=request.message,
seed_docs=plan.docs,
seed_docs=plan.suggested_docs, # phase 118 (A4): the suggestion tier
settings=settings,
holder=holder,
history=hist, # phase 74: the same trimmed prior turns
@@ -804,30 +845,33 @@ async def chat(
# 4. Durable record + required per-turn log line (PLAN §9).
# Phase 37: the agent's read documents join the
# retrieval's — deduped by (source, path), order
# preserved (the read doc stays last). Phase 113:
# the retrieval now arrives in two tiers — the cited
# docs (``plan.docs``) and the related docs
# (``plan.related_docs``, the scored-but-below-the-bar
# documents). The DURABLE record keeps the full
# preserved (a read doc not already suggested stays
# last). Phase 118: the retrieval arrives in two
# tiers — the suggested docs
# (``plan.suggested_docs``, the summary-seeded
# "start here" tier, no floor, A3) and the related
# docs (``plan.related_docs``, rank 6+ after the
# suggested set). The DURABLE record keeps the full
# retrieval (LOCKED A3: query_log records retrieval,
# not citations — even on deflected turns, where the
# weak hits live in the related tier). Phase 112:
# weak hits are suggested). Phase 112/118:
# done.sources is the CITATION surface — it carries the
# cited docs + the agent-read docs on grounded turns
# and [] on deflected ones (a deflected answer cites
# nothing; the weak hits stay in the durable record).
# suggested docs + the agent-read docs (deduped,
# LOCKED A4) on grounded turns and [] on deflected
# ones (a deflected answer cites nothing; the weak
# hits stay in the durable record).
# A cancelled turn (the generator closed by the
# consumer) never reaches this step — no query_log row.
cited_docs: list[Document] = []
cited_seen: set[tuple[str, str]] = set()
for doc in [*plan.docs, *holder.read_docs]:
for doc in [*plan.suggested_docs, *holder.read_docs]:
key = (doc.source, doc.path)
if key not in cited_seen:
cited_seen.add(key)
cited_docs.append(doc)
record_docs: list[Document] = []
seen: set[tuple[str, str]] = set()
for doc in [*plan.docs, *plan.related_docs, *holder.read_docs]:
for doc in [*plan.suggested_docs, *plan.related_docs, *holder.read_docs]:
key = (doc.source, doc.path)
if key not in seen:
seen.add(key)
@@ -852,7 +896,8 @@ async def chat(
logger.exception("chat: failed to write query_log question=%r", request.message)
logger.info(
"question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d tuning=%d "
"question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d "
"suggested=%d tuning=%d "
"kb_chars=%d history_msgs=%d threshold=%.2f deflected=%s sources=%r "
"thinking_chars=%d tool_calls=%d total_ms=%d retries=%d scaffold_stripped=%d",
request.message,
@@ -860,6 +905,7 @@ async def chat(
plan.top_score,
plan.fts_hits,
plan.summary_hits,
len(plan.suggested_docs), # phase 118: the seeded suggestion tier size
plan.tuning_count,
plan.kb_chars,
len(hist),
@@ -879,14 +925,15 @@ async def chat(
# citations). The retrieval stays durably recorded above
# (query_log.sources + the log line — observability
# unchanged).
# Phase 113 (A2/A4): done.related carries the related
# tier — the scored documents that did not clear the
# usefulness bar (deduped against the cited list, the
# same (source, path) pattern as cited_docs: an agent-
# read related doc is a citation, never a "nearby doc"
# — and capped by related_max_docs in the tiering). The
# UI renders it as the de-emphasized related-docs row,
# never a citation chip; old clients ignore the field.
# Phase 118 (A3/A4): done.related carries the related
# tier — the ranked documents beyond the suggested set
# (rank 6+ after the contiguous top-N suggestion, capped
# by related_max_docs in the tiering), deduped against
# the cited list, the same (source, path) pattern as
# cited_docs: an agent-read related doc is a citation,
# never a "nearby doc". The UI renders it as the
# de-emphasized related-docs row, never a citation chip;
# old clients ignore the field.
cited_refs: list[SourceRef] = []
if not plan.deflected:
cited_refs = [
+45
View File
@@ -111,6 +111,9 @@ class Settings(BaseSettings):
# --- RAG tuning ---
embedding_dim: int = 768 # verified against aipi /v1 (embed model)
#: Phase 118 retired the full-text seeding role (A6); the suggested
#: tier (``select_suggested``) seeds the prompt now — kept for env
#: back-compat (no ``app/`` consumer left).
top_n_docs: int = 2
# Honesty gate (A8, re-tuned 2026-08-21): the ``embed`` model's cosine
# scores compress into 0.41–0.84 on the real corpus, so the old 0.30
@@ -126,6 +129,9 @@ class Settings(BaseSettings):
# ``relevance_threshold`` (a floor above the threshold is a typo that
# would make every FTS hit require a HIGH cosine anyway).
lexical_support_floor: float = 0.35
#: Phase 118 retired the full-text seeding role (A6); the suggested
#: tier (``select_suggested``) seeds the prompt now (no floor, A3) —
#: kept for env back-compat (no ``app/`` consumer left).
#: Usefulness bar for the citation slot (phase 113, LOCKED A2): a
#: retrieved document earns ``done.sources`` (the UI's citation chip)
#: only when the **cosine** of its best hit chunk clears this floor —
@@ -149,6 +155,26 @@ class Settings(BaseSettings):
#: docs at all (the kill switch); a negative value fails startup
#: loudly (the ``agent_max_rounds`` pattern).
related_max_docs: int = 2
#: Cap on the "start here" suggestion tier (phase 118, LOCKED A3 —
#: the owner directive, TODO L3): a grounded turn seeds the top-N
#: related documents into the prompt as SUMMARY blocks (opt-in
#: starting points, never citations) and the LLM extends its context
#: by reading only what it needs. NO cosine floor applies — unlike
#: the cited tier's ``source_usefulness_floor``, a lexical-only hit
#: (cosine 0.0) is a valid starting point when it ranks. Default 5;
#: tunable via ``BOR_SUGGESTED_DOCS``. A value below 1 is a typo —
#: the validator fails startup loudly (the ``agent_max_rounds``
#: pattern).
suggested_docs: int = 5
#: Preview cap for a suggestion block whose document summary is missing
#: (phase 118, task 03, LOCKED A5): a NULL/blank ``doc.summary`` (a
#: fail-soft import miss) falls back to the first ``suggestion_preview_chars``
#: characters of the document content plus the shared
#: ``[…truncated…]`` marker — deterministic, no LLM call at chat time.
#: Tolerates content at or under the cap whole (no marker — nothing was
#: cut). ``0``/negative is a typo (empty preview) — the validator fails
#: startup loudly (the ``agent_max_rounds`` pattern).
suggestion_preview_chars: int = 400
#: Maximum output tokens a chat answer may use (owner instruction
#: 2026-08-22: answers must run to their natural end — the old hard
#: 700-token cap cut long answers off mid-sentence).
@@ -387,6 +413,25 @@ class Settings(BaseSettings):
raise ValueError("related_max_docs must be >= 0 (0 = no related docs)")
return v
@field_validator("suggested_docs")
@classmethod
def _suggested_docs_at_least_one(cls, v: int) -> int:
"""The suggestion tier always seeds at least one summary block —
``0`` (no starting points) and negatives are typos (the
``agent_max_rounds`` pattern, phase 118)."""
if v < 1:
raise ValueError("suggested_docs must be >= 1")
return v
@field_validator("suggestion_preview_chars")
@classmethod
def _suggestion_preview_chars_positive(cls, v: int) -> int:
"""``0``/negative would preview an empty/absent prefix — fail loud at
startup (the ``agent_max_rounds`` pattern, phase 118)."""
if v <= 0:
raise ValueError("suggestion_preview_chars must be > 0 (chars)")
return v
@field_validator("import_extensions")
@classmethod
def _import_extensions_known(cls, v: str) -> str:
+5 -4
View File
@@ -133,10 +133,11 @@ class Document(Base):
Boolean, default=False, server_default=text("false"), nullable=False
)
#: Lite-model summary, phase 30. Natural-language summary of the
#: document (non-markdown A9 docs only, generated at import time by the
#: aipi ``lite`` model). NULL for markdown docs, pre-phase-30 rows, and
#: the fail-soft path where summary generation failed but the document
#: was still indexed.
#: document (every A9 doc — markdown included since phase 118, A2 —
#: generated at import time by the aipi ``lite`` model). NULL for
#: pre-phase-30 rows, the fail-soft path where summary generation
#: failed, and until the phase-118 backfill stores one on the next
#: sync.
summary: Mapped[str | None] = mapped_column(Text, default=None)
chunks: Mapped[list[Chunk]] = relationship(
+44 -32
View File
@@ -97,12 +97,13 @@ task 04):
context; ``grep`` — which searches the whole document — is the
follow-up), with the truncation recorded on the holder so the loop
yields a :class:`app.rag.llm.ToolResultPiece` (task 02 → the SSE
``tool_result`` frame + UI marker). **A7 scope clarification:** the
never-truncated contract is for the retrieval ``<documents>`` path
(the top-2 seed documents stay whole — "this should never happen");
the ``read`` TOOL path is the only capped read, per the owner's
explicit request — the two paths are distinct (retrieval seeds vs.
agent-requested additions). And ``grep`` greps the
``tool_result`` frame + UI marker). **A6 re-revised contract
(phase 118, owner directive 2026-09-15):** the retrieval
``<documents>`` path seeds SUMMARIES only — a suggested document's
full text never enters the prompt on the retrieval path; full text
enters the context ONLY through this ``read`` TOOL path, which is
the only capped read (the phase-95 cap unchanged). And ``grep``
greps the
indexed documents (or the one document a combined ``source/path``
names) for a case-insensitive fixed substring and returns up to 20
``source/path:line: text`` match lines (owner-locked A5, phase 68),
@@ -138,7 +139,9 @@ task 04):
:data:`NOT_A_FOLDER`, the drill-down teaching with the argument
echoed and the deepest existing ancestor's direct subfolders
listed, so the model self-corrects in the next round; a document
already in context (seed or previously read) →
already READ into full-text context (phase 118: the suggested
seeds are summary blocks in the prompt, not full text — only an
already-read document is refused) →
:data:`ALREADY_IN_CONTEXT` (phase 72, task 05 gate iteration:
the line names the correct action — answer from the text already
in the prompt, do not call read again — so a fired refusal ends
@@ -301,19 +304,21 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"function": {
"name": "read",
"description": (
"Do not call this tool for a document already shown in "
"the <documents> section, even when the user asks you to "
"open or read it — its full text is already in your "
"prompt; answer directly from it. Use it only to add a "
"document NOT already in <documents> to your context, "
"by its combined `source/path` string. Very large "
"documents are truncated: you receive the first part "
"plus a TRUNCATED notice naming how many more characters "
"exist — the notice is authoritative, the document did "
"NOT end where it stopped. Follow it and use `grep` "
"(pattern) to locate the rest — it searches the whole "
"document. Call one tool at a time — wait for this "
"result before your next call."
"The <documents> section shows the SUMMARIES of the "
"top-ranked documents — their full texts are NOT in "
"your prompt yet. Use this tool to add one of them (or "
"any other document) to your context, by its combined "
"`source/path` string, exactly as shown in the `ls` "
"output or the <documents> blocks. Do not re-read a "
"document you have already read — its full text is "
"already in your prompt. Very large documents are "
"truncated: you receive the first part plus a TRUNCATED "
"notice naming how many more characters exist — the "
"notice is authoritative, the document did NOT end "
"where it stopped. Follow it and use `grep` (pattern) "
"to locate the rest — it searches the whole document. "
"Call one tool at a time — wait for this result before "
"your next call."
),
"parameters": {
"type": "object",
@@ -326,10 +331,9 @@ AGENT_TOOLS: list[dict[str, Any]] = [
"shown in the `ls` output (e.g. "
"'homelab/active/container_caddy/caddy.md'). "
"A bare document path (without the source "
"name) will not resolve. Only pass a document "
"NOT already shown in the <documents> "
"section — it is already in your context; do "
"not re-read it."
"name) will not resolve. Do not re-read a "
"document you have already read — its full "
"text is already in your prompt."
),
}
},
@@ -1152,10 +1156,15 @@ def _execute_tool(
arg = raw_path.strip() if isinstance(raw_path, str) else ""
if not arg:
return MISSING_READ_ARGS
known = {(doc.source, doc.path) for doc in (*seed_docs, *holder.read_docs)}
# Phase 118 (A6): the dedupe set is ``holder.read_docs`` ONLY —
# the ``seed_docs`` are SUMMARY blocks in the prompt, not full
# text, so a FIRST read of a suggested document adds its full
# text through the path below; only a document ALREADY READ is
# refused.
known = {(doc.source, doc.path) for doc in holder.read_docs}
# The dedupe check needs no DB: the split pair of a combined
# identity that is in context is in `known` as-is (the resolve
# below would find the same document).
# identity that is already in full-text context is in `known`
# as-is (the resolve below would find the same document).
if "/" in arg:
src, _, p = arg.partition("/")
if (src, p) in known:
@@ -1326,11 +1335,14 @@ async def run_agent(
the capture mechanism for new registry entries; *holder* accumulates
the turn's ``scaffold_stripped`` total for the API layer's log line.
``seed_docs`` are the documents the retrieval already put in context
(they shape the *system_prompt* the caller built); re-reading one of
them is rejected with :data:`ALREADY_IN_CONTEXT` (the phase-72
teaching line — answer from the text already in the prompt) — the
rejection counts in nothing, but it still consumes a round.
``seed_docs`` are the suggested documents whose SUMMARY blocks the
caller put in the *system_prompt* (phase 118: the retrieval seeds
summaries, never full texts); reading one of them ADDS its full
text to the context through the ordinary ``read`` path — only a
document ALREADY READ is rejected with :data:`ALREADY_IN_CONTEXT`
(the phase-72 teaching line — answer from the text already in the
prompt); the rejection counts in nothing, but it still consumes a
round.
DB sessions (SEC-14-04): *db_factory* is a callable that returns a
new :class:`sqlalchemy.orm.Session` (e.g. ``lambda: SessionLocal()``).
+62 -12
View File
@@ -10,7 +10,8 @@ the two-phase upsert:
2. embed the new chunks in batches and attach the vectors
3. commit — one transaction per file, so a failed embedding leaves the
database untouched and the file is simply retried on the next run
4. non-markdown files only (phase 30): generate a ``lite``-model summary
4. every file (phase 30; phase 118, A2: markdown included — the
non-markdown-only scope is retired): generate a ``lite``-model summary
and, best-effort, store it on ``documents.summary`` plus one extra
embedded chunk (``is_summary``, position −1). The document row and its
content chunks are already committed at this point, so a summary
@@ -42,6 +43,17 @@ guard) and counted in ``summary.dates_updated`` — unless the row
carries the owner's manual correction (``created_at_manual``, D1), which
the sync never touches.
NULL-summary backfill (phase 118, A2): an UNCHANGED file (same
``content_hash``) whose ``documents.summary`` is still NULL — a
pre-phase-30 row, or an earlier fail-soft miss — gets the same
best-effort summary pass on every sync until it sticks. A success counts
``summary_backfilled`` (never ``summaries``) and touches nothing else: no
content re-embed, no added/updated/pruned count — so no
``sources_meta`` bump, no KB-overview/folder-summary regeneration. The
backfill runs BEFORE the ``created_at_manual`` early-return (the manual
flag protects the DATE only, D1) and the strict ``is None`` check leaves
owner-set summaries (even empty strings, phase 57) alone.
``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now.
"""
@@ -99,12 +111,18 @@ class ImportSummary:
errors: int = 0
chunks: int = 0
embed_batches: int = 0
#: Non-markdown files whose lite summary was generated + indexed
#: (phase 30). One ``is_summary`` chunk per success.
#: Files whose lite summary was generated + indexed (phase 30; phase
#: 118, A2: every A9 format, markdown included). One ``is_summary``
#: chunk per success.
summaries: int = 0
#: Non-markdown files whose summary generation failed (best-effort —
#: the document is still indexed, without a summary).
#: Files whose summary generation failed (best-effort — the document
#: is still indexed, without a summary).
summary_errors: int = 0
#: Unchanged docs whose NULL summary was backfilled (phase 118, A2) —
#: one ``is_summary`` chunk per success; the content is untouched, so
#: a backfill NEVER counts added/updated/pruned (no
#: ``sources_meta`` bump, no overview/folder-summary regeneration).
summary_backfilled: int = 0
#: Files whose ``created_at`` was refreshed on the UNCHANGED path —
#: content untouched, date re-sourced (phase 106, D4: the date may
#: go OLDER; a date-only refresh NEVER counts added/updated/pruned,
@@ -125,7 +143,7 @@ class ImportSummary:
logger.info(
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
"dates_updated=%d formats=%s",
"summary_backfilled=%d dates_updated=%d formats=%s",
self.files,
self.added,
self.updated,
@@ -136,6 +154,7 @@ class ImportSummary:
self.embed_batches,
self.summaries,
self.summary_errors,
self.summary_backfilled,
self.dates_updated,
self.format_counts(),
)
@@ -442,6 +461,20 @@ async def _index_file(
if doc is not None and doc.content_hash == digest:
summary.unchanged += 1
logger.info("import: unchanged source=%s path=%s", source, rel)
# Phase 118 (A2): an unchanged doc whose summary is still NULL
# (a pre-phase-30 row, or an earlier fail-soft miss) gets a
# summary-only backfill — one ``is_summary`` chunk, no content
# re-embed, and NEVER an added/updated/pruned count (so no
# ``sources_meta`` bump, no overview/folder-summary
# regeneration). Strict ``is None``: an empty-string summary is
# owner-set (phase 57) and is never overwritten. BEFORE the
# manual-date early-return: ``created_at_manual`` protects the
# DATE only (phase 106, D1), not the summary.
if doc.summary is None:
await _store_summary(
session, doc=doc, source=source, rel=rel, content=content,
llm=llm, summary=summary, backfill=True,
)
if doc.created_at_manual:
# D1/D4: the owner's correction survives the sync — no
# write at all (the phase-97 ``manually_edited`` precedent).
@@ -536,10 +569,11 @@ async def _index_file(
summary.chunks += len(chunks_text)
logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, len(chunks_text))
# Phase 30: markdown is already natural language, so only the other A9
# formats (txt, yaml, yml, json, py) get a ``lite``-model summary.
if full_path.suffix.lower() in (".md", ".markdown"):
return
# Phase 30; phase 118 (A2, 2026-09-15): EVERY new/changed document
# gets a ``lite``-model summary — markdown included. Phase 30's
# "markdown is already natural language" exclusion is retired: the
# summary is the retrieval seed context (the phase-118 suggestion
# blocks), not a formatting convenience.
await _store_summary(
session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary
)
@@ -554,6 +588,7 @@ async def _store_summary(
content: str,
llm: Embedder,
summary: ImportSummary,
backfill: bool = False,
) -> None:
"""Best-effort ``lite`` summary for one already-committed document.
@@ -569,6 +604,11 @@ async def _store_summary(
:class:`EmbeddingError` only rolls back the summary rows — the file
stays indexed, without a summary, and the failure is counted in
``summary_errors`` (PLAN phase 30).
``backfill`` (phase 118, A2): the unchanged-doc NULL-summary path —
a success counts ``summary_backfilled`` instead of ``summaries``
(the doc content is untouched, so the import's KB-change signal must
not move); the rest of the mechanics are identical.
"""
try:
text = await generate_summary(llm, source=source, path=rel, content=content)
@@ -588,8 +628,18 @@ async def _store_summary(
# ``expire_on_commit=False`` — reflects the committed state.
doc.chunks.append(chunk)
session.commit()
summary.summaries += 1
logger.info("import: summary source=%s path=%s chars=%d", source, rel, len(text))
if backfill:
# Phase 118 (A2): the backfill counts itself apart from fresh
# imports — the doc content is unchanged, so ``summaries``
# (a KB-change signal) must not move.
summary.summary_backfilled += 1
logger.info(
"import: summary-backfill source=%s path=%s chars=%d",
source, rel, len(text),
)
else:
summary.summaries += 1
logger.info("import: summary source=%s path=%s chars=%d", source, rel, len(text))
except (LLMError, EmbeddingError) as e:
session.rollback()
summary.summary_errors += 1
+3 -2
View File
@@ -1,8 +1,9 @@
"""Async OpenAI-compatible client for the self-hosted aipi endpoint (PLAN A5).
Provides the embeddings surface (importer, retrieval), one-shot chat
completions (phase 30: the ``lite`` model summarizes non-markdown
documents at import time), and chat streaming (PLAN A15) for the RAG
completions (phase 30: the ``lite`` model summarizes documents at import
time — every A9 format, markdown included since phase 118), and chat
streaming (PLAN A15) for the RAG
pipeline. Chat streaming yields typed :class:`StreamPiece` values
(phase 17) and — when the caller passes a ``tools`` list —
:class:`ToolCallPiece` values (phase 37): aipi's ``turbo`` model streams
+3 -2
View File
@@ -100,8 +100,9 @@ def build_overview_prompt(
* ``user`` — one line per document,
``source — path — title — {first line of summary}``, joined with
newlines. The summary field is omitted when the document has no
summary (markdown docs and the fail-soft path — no dangling
dash). The list is capped at *max_chars* (default
summary (pre-phase-30 rows, the fail-soft path, and pre-backfill
NULL rows — no dangling dash). The list is capped at *max_chars*
(default
``BOR_OVERVIEW_INPUT_MAX_CHARS``): overflow is cut exactly at the
cap and the shared ``[…truncated…]`` marker is appended on its own
line, so the model never sees more than the cap and the cut is
+99 -41
View File
@@ -7,7 +7,14 @@ no mandated deflection opening; the honesty gate itself is unchanged.)
Two modes:
* ``HIGH`` — grounded turn: full top-document texts under ``<documents>``.
* ``HIGH`` — grounded turn: the top-ranked documents' SUMMARIES under
``<documents>`` (phase 118, LOCKED A6 re-revising A7 — the "start here"
suggestion seeding: each block is the document's stored summary, never
the full content; the LLM extends its context by ``read``-ing a
document's full text through the capped ``read`` tool, the ONLY full-
text path. A NULL/blank summary — a fail-soft import miss — falls back
to a ``suggestion_preview_chars`` content preview + the shared
``[…truncated…]`` marker; no LLM call at chat time, LOCKED A5).
* ``LOW`` — deflection turn: weak-hit *titles only* plus the
``DEFLECT_MODE`` marker (the E2E mock LLM keys on that marker).
@@ -124,6 +131,24 @@ _KB_INTRO = (
"(generated at import time):\n"
)
#: The ``<documents>`` section's start-here intro (phase 118, task 03 —
#: the owner directive, TODO L3): the seeded blocks are the SUMMARIES of
#: the top-ranked documents for the question, opt-in starting points —
#: start here if one seems right; ``read`` the document's combined
#: ``source/path`` to add its FULL content before answering specifics
#: (the full text is NOT in the prompt until you read it); cite the
#: document(s) you used, by path. Rendered BEFORE the first block, only
#: when at least one block is present. The ``<documents>`` tag and the
#: per-block ``<document>`` markup stay byte-stable around it (the E2E
#: mock's block parser and the ``read``-tool copy key off both).
SUGGEST_INTRO = (
"The blocks below are the summaries of the top-ranked documents for "
"your question — start here if one seems right to you: call `read` "
"with that document's combined `source/path` to add its full content "
"before answering specifics (its full text is not in the prompt until "
"you read it). Cite the document(s) you used, by path."
)
#: The ``<tools>`` instructions section — **HIGH prompt only** (phase 37,
#: task 03; phase 70: the copy is rewritten for the harness-aligned
#: ``ls`` / ``read`` / ``grep`` shapes, names/args exactly as the
@@ -158,12 +183,15 @@ _KB_INTRO = (
#: the user's "open it / read it" and ``read`` seed-context documents,
#: then repeated the refused call; run 2: 8/18, 44% — the repeat is
#: gone, but a grep turn batched two calls per reply and the harness
#: runs only the first of a batch): the do-not-read rule names the
#: user-command scenario (a document already in the ``<documents>``
#: section: do not call ``read``, answer from the text already in the
#: prompt — an anchor on the concrete ``<document path="...">`` markup
#: was tried and REVERTED: it primed the model to latch the seed
#: documents' paths as ``ls`` scopes, regressing the incident turn);
#: runs only the first of a batch): the ``read`` clause carries the
#: phase-118 summary-seed contract (the ``<documents>`` section holds
#: SUMMARIES — a suggested document's full text is not in the prompt
#: until ``read`` adds it; do not re-read a document you have already
#: read — answer from the text already in the prompt; if the user
#: asks to open or read a suggested document, ``read`` it — that is
#: the point of the section — the phase-72 do-not-read rule and the
#: reverted ``<document path="...">`` anchor are retired with the
#: full-text seeds);
#: the one-call rule names the consequence (a batched second call is
#: discarded — runs only the first); the never-repeat rule says why
#: (the refusal already told you the correct form); the ``grep`` clause
@@ -171,10 +199,11 @@ _KB_INTRO = (
#: live runs showed the model scoping ``grep`` with an ``ls``-style
#: source name — the incident shape, but on grep). The behavioral
#: contract lives in the ``AGENT_TOOLS`` descriptions as well (the most
#: local text at call time): ``read`` must not be called for a
#: ``<documents>`` document at all; ``grep`` with only ``pattern``
#: searches the whole knowledge base, and a source name is not a
#: document.
#: local text at call time): ``read`` adds a document's FULL text by
#: its combined ``source/path`` identity — the ``<documents>``
#: summaries are the starting points, not the content; ``grep`` with
#: only ``pattern`` searches the whole knowledge base, and a source
#: name is not a document.
TOOLS_SECTION: str = (
"<tools>\n"
"You may extend your context with three tools. `ls` lists the "
@@ -189,12 +218,16 @@ TOOLS_SECTION: str = (
"`source: X | path: Y | title: Z`; to find one specific document "
"without listing, use `grep`. `read` pulls in one document by its "
"combined `source/path` string, exactly as shown in the `ls` "
"output — including the source name — adding its full content to "
"your context. Do not call `read` for a document already shown in "
"the <documents> section, even when the user asks you to open or "
"read it — its full text is already in your prompt; answer "
"directly from it. For `read`, a bare document path (without the "
"source name) will not resolve. Very large documents are capped: a "
"output — including the source name — or in the <documents> "
"summary blocks — adding its full content to your context. The "
"<documents> section holds SUMMARIES — the full text of a "
"suggested document is not in your prompt until you `read` it. Do "
"not re-read a document you have already read — its full text is "
"already in your prompt; answer directly from it. If the user "
"asks you to open or read a suggested document, `read` it — that "
"is the point of the section. For `read`, a bare document path "
"(without the source name) will not resolve. Very large documents "
"are capped: a "
"cut read returns the first part plus a TRUNCATED notice — the "
"document did not end where it stopped; use `grep` (pattern) to "
"find the rest, it searches the whole document. `grep` locates an exact string "
@@ -355,17 +388,42 @@ def build_kb_section(overview: str, max_chars: int | None = None) -> str:
return ""
def _document_body(doc: Document) -> str:
"""The body of one ``<document>`` suggestion block (phase 118, LOCKED
A6): the document's stored summary (stripped) — NEVER the full
content.
Defensive fallback ONLY when the summary is missing (``None`` or
whitespace — a fail-soft import miss, LOCKED A5): the first
``suggestion_preview_chars`` characters of the content plus the
shared :data:`TRUNCATION_MARKER` on its own line — the settings read
happens on this fallback path ONLY (a prompt built from
summary-bearing docs makes no settings call for the cap). Content at
or under the cap rides whole, unmarked (nothing was cut). No LLM
call at chat time — the preview is deterministic.
"""
summary = (doc.summary or "").strip()
if summary:
return summary
limit = get_settings().suggestion_preview_chars
content = doc.content
if len(content) > limit:
return content[:limit] + "\n" + TRUNCATION_MARKER
return content
def build_high_prompt(
documents: Sequence[Document],
notes: Sequence[str] | None = None,
kb_overview: str | None = None,
) -> str:
"""Grounded turn: locked persona (+ steering, + KB overview) + full
texts of the top documents + the ``<tools>`` instructions (phase 37;
phase 70: the harness-aligned ``ls`` / ``read`` / ``grep`` shapes;
phase 72: the copy states the document-identity contract — the
source-name ``ls`` scope, the combined ``source/path`` identity for
``read``/``grep`` — up front).
"""Grounded turn: locked persona (+ steering, + KB overview) + the
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
(phase 37; phase 70: the harness-aligned ``ls`` / ``read`` /
``grep`` shapes; phase 72: the document-identity contract up front;
phase 118, LOCKED A6 re-revising A7: the ``<documents>`` section
seeds SUMMARIES, never full texts — full text enters the context
only through the capped ``read`` tool).
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their
@@ -373,42 +431,42 @@ source-name ``ls`` scope, the combined ``source/path`` identity for
cap — not the prompt — decides whether the tools are actually
offered to the model, see :mod:`app.rag.agent`).
When at least one block is present, the section leads with the
start-here :data:`SUGGEST_INTRO` line (before the first block — the
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent): the
blocks are the summaries of the top-ranked documents, ``read`` adds
the full text, and the answer cites the document(s) used by path.
This is NOT the reverted phase-72 in-context reminder (that copy
taught the seed documents as already-read context; this one teaches
the summary-as-starting-point contract the A6 re-revision requires).
Each ``<document>`` block carries the identity attributes
``source`` / ``path`` / ``title`` — and, since phase 106 (D5),
``date`` (the row's ``created_at`` UTC date part, ``YYYY-MM-DD``,
APPENDED after ``title`` — the only position; always present,
``created_at`` is NOT NULL) — plus the document's full text.
Gate-iteration note (task 05, 2026-09-03/04): an in-context reminder
LEADING this section (the document texts are already context — do
not ``read`` one the user asked to open) was tried and REVERTED:
it never flipped the seed-doc reads (15/15 across gate runs 1-5)
and correlated with the incident-turn regression (the model latched
the seed documents' paths as ``ls`` scopes — cap reached on the
"list the files in this directory" turn) whenever the copy named
the ``<document>`` blocks explicitly.
``created_at`` is NOT NULL) — plus the document's summary
(:func:`_document_body`; the NULL/whitespace-summary preview
fallback is LOCKED A5).
"""
# 2026-09-04 (controlled tool-calling fast loop): the do-not-read
# rule for seed documents lives in TOOLS_SECTION and the ``read``
# tool descriptions (the copy levers that stuck — see the gate's
# telemetry in TOOL_CALLING_TESTING.md). A per-block instruction
# attribute at the ``source``/``path`` copy site was TRIED and
# REVERTED the same day (no improvement across runs; the block stays
# exactly the document identity + full text).
# Phase 106 (D5): every document the model sees carries its
# creation date — the block's ``date`` attribute (the row's
# ``created_at`` UTC date part, appended after ``title`` — the
# only position; always present, ``created_at`` is NOT NULL).
# Phase 118 (A6): the block body is the summary — the full content
# stays out of the prompt (the capped ``read`` tool is the only
# full-text path).
blocks = [
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
f'date="{doc.created_at:%Y-%m-%d}">\n'
f"{doc.content}\n"
f"{_document_body(doc)}\n"
"</document>"
for doc in documents
]
body = "\n\n".join(blocks) if blocks else (
"(no documents matched — do not invent specifics)"
)
if blocks:
body = SUGGEST_INTRO + "\n\n" + body
prompt = _base("HIGH")
for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])):
if part:
+113 -13
View File
@@ -35,12 +35,16 @@
ranked documents (up to ``BOR_RELATED_MAX_DOCS``) become the related
tier.
The product requirement (LOCKED A7, revised 2026-08-24): the LLM receives
the **entire relevant document**, not just the chunk — chunk hits map back
to their parents, dedupe, rank by best fused score, and the full text of
the top-N documents is always fed through, never truncated. If a future KB
ever makes the prompt too large for the model, the ``LLMError`` → SSE
``error`` path surfaces it loudly — no silent partial context.
The product requirement (A7, re-revised by the phase-118 owner directive,
LOCKED A6, 2026-09-15): the retrieval path seeds **summaries** — the
suggestion tier (:func:`select_suggested`, top-N distinct documents, no
cosine floor, LOCKED A3) whose summary blocks are the grounded prompt's
``<documents>`` starting points. The full text of a document enters the
context ONLY through the agent's capped ``read`` tool
(:mod:`app.rag.agent`; ``BOR_READ_MAX_CHARS`` + :data:`TRUNCATION_MARKER`),
never through the retrieval seeding. A document's content itself is still
carried on its rows byte-identical — the ``read`` tool serves it whole,
un-truncated up to its cap.
Deterministic tie-break for equal fused scores:
``(−fused, −cosine, document.path, chunk.position)``.
@@ -61,9 +65,12 @@ from sqlalchemy.orm import Session
from app.config import get_settings
from app.models import Chunk, Document
#: Shared overflow marker — now used by the steering (<tuning>) section
#: only (phase 15; imported by ``app.rag.prompts``). The document context
#: path never truncates (A7 revised, owner permission 2026-08-24).
#: Shared overflow marker (phase 15; imported by ``app.rag.prompts``)
#: — used by the steering (<tuning>) section, the phase-118 NULL-summary
#: suggestion preview fallback (A5), and the capped agent ``read``
#: result. The seeded summary blocks and the ``read``-served document
#: content never truncate silently (A6 re-revised): full text enters the
#: context only through the capped ``read`` tool.
TRUNCATION_MARKER = "[…truncated…]"
#: Alphanumeric tokens of a question (``to_tsquery`` input, OR-joined),
@@ -574,6 +581,11 @@ def select_documents_tiered(
"""Tier chunk hits into the cited and the related parent documents
(phase 113, LOCKED A2/A4 — the usefulness bar).
Phase 118 retired the full-text seeding role (A6); the suggested
tier (:func:`select_suggested`) seeds the prompt now — this helper
stays as a dormant public helper (env back-compat for the settings
it was calibrated by).
Distinct parent documents are ranked exactly like :func:`select_documents`
(best fused score first — the same stable score-descending walk, so a
document's rank position is fixed by its FIRST seen chunk) and each
@@ -603,7 +615,11 @@ def select_documents_tiered(
The returned rows carry the full document content, byte-identical —
a matched parent document is **never truncated** (A7 revised, owner
permission 2026-08-24).
permission 2026-08-24; A6 re-revised 2026-09-15: the retrieval path
seeds SUMMARIES — the cited tier's full texts no longer ride the
grounded prompt, full text enters the context only through the
capped ``read`` tool; the rows themselves still carry the whole
content).
"""
top_n = n if n is not None else get_settings().top_n_docs
no_bar = floor <= 0.0
@@ -642,12 +658,20 @@ def select_documents(
) -> list[Document]:
"""Map chunk hits to distinct parent documents, ranked by best fused score.
Phase 118 retired the full-text seeding role (A6); the suggested
tier (:func:`select_suggested`) seeds the prompt now — this helper
stays as a dormant public helper (env back-compat for the settings
it was calibrated by).
At most *n* documents are returned (default ``BOR_TOP_N_DOCS``). The
returned rows carry the full document content, byte-identical — a
matched parent document is **never truncated** (A7 revised, owner
permission 2026-08-24). There is deliberately no context budget: an
oversized prompt must fail loudly through the ``LLMError`` → SSE
``error`` path, never arrive as silent partial context.
permission 2026-08-24; A6 re-revised 2026-09-15: the seeded prompt
now carries SUMMARIES — the full text reaches the context only
through the capped ``read`` tool, not through this selection). There
is deliberately no context budget: an oversized prompt must fail
loudly through the ``LLMError`` → SSE ``error`` path, never arrive
as silent partial context.
Phase 113: a thin wrapper on :func:`select_documents_tiered` — the
legacy "any score, top-N" behavior is the cited tier with a zero
@@ -656,3 +680,79 @@ def select_documents(
"""
cited, _ = select_documents_tiered(chunks, n, 0.0, 0)
return cited
def select_suggested(
chunks: Sequence[RetrievedChunk],
n: int | None = None,
) -> list[Document]:
"""Top-N distinct parent documents in fused rank order — the phase-118
"start here" suggestion tier (LOCKED A3), with NO cosine floor.
Distinct parent documents are walked in the SAME stable score-
descending order as :func:`select_documents_tiered` (a document's rank
position is fixed by its FIRST seen chunk; dedupe by ``document.id``),
and at most *n* of them are returned (default the
``BOR_SUGGESTED_DOCS`` setting, 5). Unlike the phase-113 cited tier,
the usefulness bar NEVER filters here: a lexical-only hit with
cosine 0.0 is suggested when it ranks. Suggestions are opt-in
starting points, not citations — the seeded prompt carries the
document's summary, and the LLM decides whether to extend its context
by reading the document's full text.
The returned rows carry the full document content, byte-identical —
the content is what the agent's ``read`` tool serves later (never
truncated; A6 re-revises A7: full text enters the context only through
the capped ``read`` tool).
"""
top_n = n if n is not None else get_settings().suggested_docs
order: list[Document] = []
seen: set[uuid.UUID] = set()
for rc in sorted(chunks, key=lambda c: c.score, reverse=True):
if len(order) >= top_n:
break
doc = rc.document
if doc.id in seen:
continue
seen.add(doc.id)
order.append(doc)
return order
def select_related(
chunks: Sequence[RetrievedChunk],
excluded_ids: set[uuid.UUID],
cap: int,
) -> list[Document]:
"""The documents ranked AFTER *excluded_ids* — the phase-118 related
tier (rank 6+ for the contiguous top-5 suggestion set), up to *cap*
(``BOR_RELATED_MAX_DOCS``).
The SAME stable score-descending walk as
:func:`select_documents_tiered` / :func:`select_suggested` (a
document's rank position is fixed by its FIRST seen chunk; dedupe by
``document.id``), skipping every document whose id is in
*excluded_ids* and admitting at most *cap* documents. There is NO
cosine floor: the related tier is the ranked remainder (a lexical-
only cosine 0.0 hit is included) — its job on the ``done`` frame is
visibility (the UI's de-emphasized "nearby docs" row), not
citation. With the turn wiring's exclusion — exactly the suggested
tier's document ids (LOCKED A3: a contiguous top-N, no floor) —
"excluding the suggested" is exactly "rank 6+".
The returned rows carry the full document content, byte-identical
(the tier is metadata for the ``done`` frame and the durable
record; the prompt and ``read`` contract are untouched).
"""
out: list[Document] = []
seen: set[uuid.UUID] = set()
for rc in sorted(chunks, key=lambda c: c.score, reverse=True):
if len(out) >= cap:
break
doc = rc.document
if doc.id in seen or doc.id in excluded_ids:
continue
seen.add(doc.id)
out.append(doc)
return out
+22 -7
View File
@@ -388,9 +388,10 @@ class DocContent(BaseModel):
path: str
title: str
format: str
#: Lite-model summary (phase 30) — non-markdown A9 docs only; None for
#: markdown documents, pre-phase-30 rows, and the fail-soft path where
#: summary generation failed but the document was still indexed.
#: Lite-model summary (phase 30) — every A9 doc, markdown included
#: since phase 118 A2; None for pre-phase-30 rows, the fail-soft path
#: where summary generation failed, and until the phase-118 backfill
#: stores one on the next sync.
summary: str | None = None
#: The document's creation date (phase 106, D8) — ISO-8601, verbatim
#: from the row; the viewer's top meta row renders the ``Created``
@@ -742,14 +743,26 @@ class ChatMessage(BaseModel):
"""One conversation record in the ``bor.chat.v1`` localStorage shape
(phase 14) — the stored ``messages`` payload of a saved chat (phase 50).
``{who, text, sources?, deflected?, suggestions?, thinking?, tools?,
stopped?}`` — raw text, never HTML, so a saved chat restores
pixel-identical through the existing ``renderStoredMessage`` path.
``{who, text, sources?, related?, deflected?, suggestions?,
thinking?, tools?, stopped?}`` — raw text, never HTML, so a saved
chat restores pixel-identical through the existing
``renderStoredMessage`` path.
``extra="forbid"`` rejects unknown keys (a corrupted or HTML-shaped
payload, e.g. a stray ``<b>``-ish extra key) at the boundary with a
422, so nothing outside this shape can poison a restored
conversation.
``related`` (phase 113, the related-doc tier the UI persists with
every grounded brain record — the restore path re-renders the
de-emphasized row from it): the same :class:`SourceRef` list shape
as ``sources``, the same cap. Its absence from this model was a
phase-113 omission — the ``extra="forbid"`` boundary 422'd every
done-time auto-save carrying the key (the A2 quiet-failure path
swallowed it), leaving grounded turns' brain messages unsaved;
pinned by ``tests/integration/test_chats_api.py`` (the full brain
record carries ``related``) and the E2E ``test_share_chat``
auto-save count.
Phase 83 (SEC-05) bounds the anonymous write surface (``POST/PUT
/api/chats`` is public — the row id is the credential, phase 55 A1):
``text`` / ``thinking`` carry :class:`HistoryTurn`'s 32 000 caps
@@ -757,7 +770,8 @@ class ChatMessage(BaseModel):
path, so a saved chat can never legitimately carry more), and the
nested lists get length caps (``max_length``) sized to the realistic
``bor.chat.v1`` record the UI produces (``sources`` ≤ 20 — top-N docs
+ agent reads; ``suggestions`` ≤ 50 chips of ≤ 200 chars; ``tools``
+ agent reads; ``related`` ≤ 20 — the related-doc tier, same ref
shape; ``suggestions`` ≤ 50 chips of ≤ 200 chars; ``tools``
≤ 50 — one entry per tool call, the round cap is 10). Only value
bounds were added — the accepted/rejected KEYS are unchanged.
"""
@@ -767,6 +781,7 @@ class ChatMessage(BaseModel):
who: Literal["user", "brain"]
text: str = Field(min_length=1, max_length=32_000)
sources: list[SourceRef] | None = Field(default=None, max_length=20)
related: list[SourceRef] | None = Field(default=None, max_length=20)
deflected: bool | None = None
suggestions: list[_Chip] | None = Field(default=None, max_length=50)
thinking: str | None = Field(default=None, max_length=32_000)