brain-of-reese/AGENTS.md

# AGENTS.md — Operating Rules for All Agents on This Repo

1. **Always read `.agents/PLAN.md` first.** It contains the architecture, the
   LOCKED DECISIONS, the UI/UX standards, the data model, and the roadmap.
   Do not design anything that contradicts it.
2. **Follow the phased execution protocol in `.agents/phases/`.** Work through
   `.agents/phases/todo/` in numeric order. When a phase is complete, move its
   directory to `.agents/phases/complete/`.
3. **Strictly adhere to the LOCKED DECISIONS** (PLAN §2). If you believe a
   LOCKED decision is wrong, stop and flag it — do not silently deviate.
4. **"One Story, One Phase":** each user story in `.agents/user_stories/`
   corresponds to a distinct execution phase, and **every** phase — story-driven
   or owner-directed — includes its own dedicated Playwright E2E test suite
   (one test file for the phase, run in isolation).
5. **"UI Structure Check":** before finalizing any UI component, verify that
   it follows the layout principles in `.agents/PLAN.md` §7 (proper container
   usage, the 72rem centered reading column — every page rides the 72rem
   container at every viewport; phase 100 supersedes §7.1's 46rem chat
   column, full-width tables on the catalogue pages — no skinny
   wasted-space lists;
   the catalogue tables (RAG, Sources, History) are bounded scroll
   panels — `max-height: min(70vh, 64rem)` — with a sticky header row;
   a clipped action label shows an ellipsis hint (`.cell-clip`) with
   the full text kept reachable via the control's aria-label
   (phase 156); an in-flight turn never paints an empty answer
   bubble — the box is created empty as the reasoning anchor and
   `.msg .bubble:empty { display: none }` hides it until the first
   answer token (phase 166); an answer's inline `[text](url)`,
   `![alt](src)` and `####`+ heading render through the ONE shared
   renderer into an allowlist-gated `a.md-link` (underlined, never a
   `javascript:`/`data:` element), a container-capped
   `img.md-img` (`max-width: 100%` — no horizontal overflow at 375px)
   and an `h5`/`h6` sized into the existing type scale — never printed
   reference text (phase 180); an SVG document paints as a
   graphic — the viewer's image block adds
   `.doc-image-img--vector` (`width: 100%`, `object-fit: contain`)
   when `format` is `svg` so a `viewBox`-only drawing spans the
   `.doc-image` card instead of the 300×150 default box, while the
   chat figure and RAG-table thumbnail keep their fixed caps and the
   bytes stay an `<img>` src (never inlined) (phase 181); the
   document viewer's page strip is
   CLASS-scoped —
   the primary class shows the strip in place of the text, the
   additive class never hides the text: Text/Pages toggle group
   (`aria-pressed`, Text default) with the strip behind it
   (phase 163); a lazily-drilled KB-tree level keeps the level the
   user is ON rendered while its node fetch runs, and its ONE
   in-flight row (`role="status"`) / alert + Retry row
   (`role="alert"`, a real `<button>`) sits inside that same bounded
   scroll panel — and the stat cards read the server's `totals`,
   never a walk of a depth-limited tree (phase 167); an answer's
   code block is EITHER verbatim-and-badged OR plainly the model's —
   never a silently-substituted hybrid: the ` ```bor ` locator fence
   renders through the ONE shared renderer as
   `.verbatim[data-bor-locator]` with the model's body DROPPED
   (`renderMarkdown(md, { verbatim: true })`, off by default so every
   other surface is byte-identical), styled from palette variables only
   with a `min(40rem, 70vh)` scroll body, a sticky `source/path`
   caption and a `user-select: none` line-number gutter, while the
   plain `.code-block` wrapper carries a Copy button and no badge
   (phase 182); those placeholders hydrate against the frame's
   `answer_text` through the ONE panel client
   (`frontend/assets/verbatim.js` — a classic script loaded AFTER
   `markdown.js` on the chat AND shared pages, no second locator grammar,
   a plain string compare against the server's canonical `locator`, the
   caption's numbers always the SERVER's extent, an unresolvable locator
   painting the pinned sentence `I could not verify this against the
   knowledge base` with no badge, no body and none of the model's bytes,
   no panel left at "Retrieving the exact text…" after a terminal frame
   (the `error` frame, the 300 s guard, Stop and the failure funnel all
   settle it), the Copy button's clipboard content equal to the server
   string exactly, and the shared page's panel Copy button + plain
   `/document.html` link the ONE owner-directed exception to that page's
   zero-controls rule) — phase 182 task 07), and the sync's peak-memory
   clause is TRAILING TEXT on the result line that already exists
   (` · peak <N> MB` on the RAG page's `#sync-progress` counts line
   or "Sync cancelled", written by the ONE `fmtSyncPeakClause` off
   `rss_peak_mb ?? rss_mb` — no new control, no new live region, no new
   element, no clause in a running label or in the failed terminal's
   empty box, and NOTHING rendered when the pair is null — phase 187
   task 03) and meets WCAG 2.1 AA basics (semantic
   landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the
   stream).
6. **"No CDN Rule":** all CSS, JS, fonts, and images must be served statically
   from within the FastAPI application. No `<script src="https://…">` or
   `<link href="https://…">` tags in HTML templates unless bundled locally.
   An integration test enforces this on every served page except
   `/doc-edit.html` (the parametrized no-CDN check in
   `tests/integration/test_api.py` — the shell, the viewer, the login and
   the shared page).
7. **"Debugpy Check":** any new entrypoint must keep the conditional import —
   `debugpy` is imported **only** when `DEBUGPY=1` (see
   `app/core/debugging.py`); default off, never imported otherwise.
8. **Git protocol:** Git is mandatory. One **atomic, professional commit per
   completed phase** (Conventional Commits, e.g. `feat(rag): stream RAG
   answers over SSE with source citations`). Always append `--no-gpg-sign`
   (the repo also sets `commit.gpgsign=false`). `.agents/` is **tracked and
   committed** with the phase — only runtime artifacts (`.agents/phase-sessions/`,
   `.agents/pipeline.log`) are gitignored.
9. **Test gates are non-negotiable:** a phase is complete only when unit +
   integration tests pass, `app/` coverage is **>90%**
   (`uv run pytest --cov=app --cov-report=term-missing`), and the phase's
   Playwright E2E file passes **in isolation**
   (`uv run pytest tests/e2e/test_<story>.py -v --no-cov`).
10. **Amply log, visibly feed back:** server-side, log the required
    per-turn line (PLAN §9 — phase 182 appends `verbatim_blocks=N` after
    `scaffold_stripped=N`: the verbatim display blocks actually SENT on
    the `done` frame, `0` on every turn without them, deflected included)
    **and** the per-response byte-budget line
    (`app/core/response_budget.py` — one
    `api method=… path=… status=… bytes=… enc=… duration_ms=…` line per
    `/api/*` response, INFO at/over `BOR_LOG_LARGE_RESPONSE_BYTES`, read it
    with `grep ' :: api '`, audit it with
    `uv run python -m scripts.api_payload_audit`, phase 167) **and** the
    sync's memory fields (phase 187 — `app/core/memory.py` reads PROCESS
    RSS, never the host's cgroup figure, and never raises: a `None`
    renders `?` through the ONE `fmt_mb` helper. `rss_mb=` is appended to
    `sync: started …` and to the importer's `import: summary …`, and
    `rss_mb=` + `rss_peak_mb=` to all three terminals (`sync: done …`,
    `sync: cancelled by user …`, and the sanitized one-line
    `sync: failed state=failed … error=…` that sits beside — never in
    place of — the `logger.exception` traceback). Append-only: no
    pre-existing field renamed, reordered or re-typed, so every older
    grep still matches; read them with
    `grep ' :: ' server.log | grep ' rss_peak_mb='`. Phase 187 task 03
    puts the SAME pair on the status: `SyncStatus.rss_mb` /
    `rss_peak_mb`, written by the ONE `_sample_rss` — forced at the run
    start, at every `phase` transition and in every terminal (which keep
    the FINAL pair beside the final counts, the phase-64 convention, and
    log exactly what the payload reports) — and throttled to one read per
    `_RSS_SAMPLE_INTERVAL_S` (5 s) from the per-file progress hook, so a
    27k-file walk costs ~600 `/proc` reads and not 27k (LOCKED A4).
    `GET /api/sync/status` reports both ADDITIVELY as `rss_mb` /
    `rss_peak_mb` (`int | null`; `null` is "no reading", never 0) and the
    failed funnel samples too, because the run that died is the run whose
    memory matters. That ONE sampler also owns the pair's invariant: the
    kernel refreshes `ru_maxrss` lazily, so a raw peak can sit BELOW the
    raw `VmRSS` (measured 164 vs 162) and `_sample_rss` lifts it to the
    current reading — every surface (both lines, the payload, the page's
    clause) therefore satisfies `peak >= current`, never a peak below the
    run's own footprint, and an unknowable peak is never fabricated from
    the current reading); UI-side,
    honor the "never stale" feedback contract (PLAN §7.4). Phase 188 adds
    the retrieval tier's ONE operator-visible line: when
    `BOR_RETRIEVAL_EF_SEARCH` sits below the A7 candidate count
    `_vector_candidates` logs `retrieval: ef_search=%d raised to %d (A7
    candidate count)` at WARNING and runs the search at the candidate
    count anyway — the clamp is never silent, and the healthy
    (ef_search >= candidate count) turn logs nothing. The value itself
    reaches Postgres as `SET LOCAL hnsw.ef_search` inside the turn's own
    transaction (LOCKED A4: no global GUC, nothing survives the request or
    a pool recycle). Task 04's live gate (the 1,000-document live-replica KB,
    the live embedding endpoint, question vectors PINNED so the access path is the
    one variable — `benchmarks/phase188_live_gate/`) found no A8, top-5 or
    `top_score` difference at any
    depth and shipped `retrieval_ef_search = 400`: at the candidate count the
    graph recovered the exact top-100 for 0.9814 of rows on average (0.94
    worst), at 400 for all of them, and NO threshold
    (`relevance_threshold`, `lexical_support_floor`, `name_hit_bonus`,
    `hybrid_top_k`, `hybrid_vector_candidates`) moved to get there (LOCKED A3).
    Two things the operator needs from the same run: building the index costs
    ~161 s per 417k chunks and ~1.6 GB (3.9 KB/row at `m = 16`, 768-dim), and
    `maintenance_work_mem = '512MB'` is NOT applicable in this container —
    `/dev/shm` is 63 MB, so a raise above that dies with `could not resize
    shared memory segment … No space left on device` and leaves an INVALID
    index for the migration's self-heal to drop and rebuild (observed live);
    the container's 64 MB default builds it fine.
11. **Never block the event loop:** anything blocking (a `subprocess`,
    synchronous file/network I/O, a heavy walk) reached from an `async`
    handler or a spawned background coroutine goes through
    `asyncio.to_thread` (or `create_subprocess_exec`). The sync's git
    calls do (phase 179: a long `git clone` used to park the single
    uvicorn loop and freeze the whole HTTP surface, the 2 s
    `GET /api/sync/status` poll included — pinned by
    `tests/integration/test_sync_git_freeze.py` with a 3 s stub and by
    `tests/e2e/test_sync_git_freeze.py` with a real ~9 s `file://`
    clone). A cooperative cancel unblocks the *run* at the `to_thread`
    await; the worker thread itself runs to completion (threads are not
    interruptible) while the loop keeps serving — pinned by the same E2E
    module's cancel-during-clone test. Since phase 186
    `scripts/git_sync.py` has **two** git spawn sites — the buffered
    `run_git` and the streaming `stream_git_lines` (the commit-date walk
    yields `git log` line by line instead of holding the whole history in
    RAM) — and **both** are off-loop via `asyncio.to_thread`; the
    `subprocess.` single-site intent survives as "one per shape, both
    owned there", enforced by
    `tests/unit/test_git_sync.py::test_only_two_git_spawn_sites`, and
    the owner-visible date contract through the stream (last-commit
    dates, the mtime fallback, the D1 manual lock) by
    `tests/e2e/test_git_log_stream_dates.py`.
12. **Never hydrate what you delete:** `Document.chunks` is
    `passive_deletes=True` (phase 183), so deleting a document removes its
    chunk rows **in the database** (`chunks_document_id_fkey` is
    `ON DELETE CASCADE`, alembic 0001) and a mass prune costs
    O(documents). The importer's prune and the source-removal endpoint both
    rely on it. Do not remove the keyword, and do not add a delete path
    that SELECTs `chunks` (or a `Chunk.embedding`) on its way to deleting
    them — one hydrated 768-dim embedding is ~24.8 KiB of Python, which is
    how the 2026-10-15 sync's 3,008-document prune reached 4 GB and locked
    up the VM. Pinned by
    `tests/integration/test_prune_cascade_statements.py` (statement count,
    zero orphan chunk rows, and a live-FK drift guard that fails the suite
    if the cascade clause ever disappears — because then
    `passive_deletes=True` would silently orphan rows). The
    `doc.chunks = [...]` **re-embed** path is the deliberate, documented
    exception (collection replacement legitimately needs the loaded
    collection; changed files only).
13. **Release the sync session at the proven-safe points only:** an import
    run holds one session and `expire_on_commit=False` (`app/db.py`) means a
    commit releases nothing, so `_unchanged_doc`'s `file_hash` stamp keeps
    every walked `Document` — content included — resident in the session's
    own strong `dirty` set (not collectable): memory grew with the KB. The
    cadence is therefore `session.expunge_all()` every
    `BOR_SYNC_SESSION_RESET_FILES` files at the TOP of the per-file loop and
    once per source before the two post-passes (phase 184; `0` = never, the
    pre-phase rollback). `expunge_all` DISCARDS unflushed changes, so never
    move it into `_index_file`, `_unchanged_doc`, `_store_summary` or a
    post-pass — their locals are live across statements — and never add a
    reset before a file's per-file commit. Pinned by
    `tests/integration/test_import_session_retention.py` (bounded with the
    cadence on, unbounded with it `0`, byte-identical summaries and DB) and
    through the UI by `tests/e2e/test_sync_session_reset.py` (two module apps,
    one per cadence: every document exactly once, the all-`unchanged` re-sync,
    and the `0`-cadence run byte-identical to the `3`-cadence one).
14. **The prune reads six columns and commits in batches:** `_prune`
    selects its candidates through
    `load_only(id, source, path, is_image, image_path, page_count)`
    `order_by(Document.id)` — the exact set the phase-122 images-off guard,
    the phase-126 parent-PDF rule and the image/page copy cleanup read, and
    NOT `content` (252 MB of markdown per full KB walk) — and commits every
    `BOR_SYNC_PRUNE_BATCH` deletes (`0` = the pre-phase single commit;
    phase 184). Never read another `Document` attribute inside that loop (a
    deferred column costs one lazy SELECT per document and unbounds the
    walk — no `raiseload` paper-over, add the column instead), and never
    swap the per-document `session.delete(doc)` for a bulk `delete()` (the
    guards, `changed_paths` and the log line depend on the per-document
    shape). A non-zero batch deliberately trades the all-or-nothing prune
    for a prefix (LOCKED A2): `seen` is recomputed from disk, so the next
    sync converges. Pinned by
    `tests/integration/test_import_session_retention.py` (the captured
    `SELECT`'s column list, exactly one document-wide `SELECT` per prune
    (a second one is a per-row lazy load), the commit counts at `10` and
    `0`, both guards on light rows) and
    `tests/e2e/test_sync_session_reset.py` (the five-document prune at batch
    `2` through the RAG page (`/sources.html`): both folders cleared, the
    survivors' chunks intact, zero orphan chunk rows).
15. **The unchanged fast path reads metadata, never markdown:** the four
    row lookups that share it (`_index_file`, `_index_pdf_file`,
    `_index_one_embedded_image`, `_index_image_file`) all go through the ONE
    statement `_doc_row_stmt` (`app/rag/importer.py`), which defers
    `Document.content` and NOTHING ELSE (phase 185, LOCKED A1) — the row
    load is the only place the fast path gets its row, and hydrating
    `content` for every inspected file is what put ~252.6 MB of markdown in
    front of a run that re-indexed 217 of ~27k files. Do not add a fifth
    whole-entity `select(Document).where(Document.source == …, Document.path
    == …)`, do not swap the `defer` for a `load_only` allowlist (a future
    column would silently become lazy and resurface as an N+1), and do not
    add `raiseload`/`noload` (LOCKED A2): `_unchanged_doc`'s phase-118
    summary-NULL backfill genuinely reads `doc.content` and must pay one
    per-row, one-time SELECT instead of raising. Pinned by
    `tests/integration/test_import_fast_path_columns.py` (the captured
    fast-path `SELECT` names no `documents.content`, exactly one `SELECT`
    per unchanged file with a summary, exactly one PK follow-up for the
    backfill), `tests/unit/test_importer_defer_content.py` (`content` is
    the only deferred column, the four loads share the helper, no
    `raiseload`/`noload` in the importer) and
    `tests/e2e/test_fast_path_deferred_content.py` (the owner-visible
    parity: the summary-NULL backfill survives the deferral with exactly
    one `is_summary` chunk, an unchanged doc's viewer shows the full
    stored text, a changed file re-indexes and the fast path still fires
    after).

16. **The cosine tier is indexed — keep it indexed, and keep `0029` the ONE
    definition of that index:** `ix_chunks_embedding_hnsw` (HNSW over
    `chunks.embedding`, `vector_cosine_ops`, `m = 16`,
    `ef_construction = 64`) exists because every turn's
    `ORDER BY embedding <=> :vec LIMIT 100` full-scanned ~3 GB on the
    interactive path (phase 188). Build and drop it ONLY through migration
    `0029` (`CREATE/DROP INDEX CONCURRENTLY` inside
    `op.get_context().autocommit_block()`, which pre-drops an INVALID remnant
    and raises if the build does not validate) — never an inline
    `CREATE INDEX` in app code, a test, or a runbook step; never a second
    vector index, a `halfvec` cast, or a rewritten `ORDER BY` (the `<=>`
    operator is what the graph serves, LOCKED A3/A6). A test that needs the
    index OFF moves the SCHEMA (`command.downgrade(cfg, "0028")` /
    `command.upgrade(cfg, "0029")`) after rolling back its own session — an
    open snapshot parks a `CONCURRENTLY` build indefinitely — and asserts the
    catalog state it claims (`pg_index.indisvalid`, `pg_get_indexdef`).
    `hnsw.ef_search` is set with `SET LOCAL` inside the turn's own transaction
    and the `int()`-coerced setting is the only source (no global `SET`, no
    pool-wide state, nothing survives a recycle); the shipped default is
    **400** because that is the depth the live gate measured exact-top-100
    parity at, so "tuning" it back to the candidate count is a recall change,
    and closing a recall gap by moving an honesty threshold is forbidden
    (LOCKED A3). The `ef_search` a turn gets is
    `max(retrieval_ef_search, hybrid_vector_candidates)` with a WARNING when
    clamped. Pinned by `tests/integration/test_chunks_hnsw_index.py` (DDL
    round trip + INVALID self-heal + drift guard),
    `tests/integration/test_chunks_hnsw_recall.py` (the tier's captured
    statement names the index, `SET LOCAL` reaches the connection, recall at
    the shipped depth with seq scans refused), and
    `tests/e2e/test_vector_index_recall.py` (the owner-visible arm: the SAME
    cited document with the index dropped and restored, the index VALID after a
    real importer run, the near-miss still deflecting in both arms, and the
    clamp read off the app's own captured stdout at `=150` / `=50`). That
    suite's sentinels are calibrated against
    `tests/fixtures/hnswkb/` — an 11-note tracked KB whose BYTES own the
    quoted cosine/lexical margins, so editing a note there moves the numbers in
    the module docstring and the pins together.

## Quick reference
```bash
podman compose up -d db                      # start Postgres 17 + pgvector
cp .env.example .env                         # once; then edit
uv run alembic upgrade head                  # apply migrations
uv run uvicorn app.main:app --reload         # dev server (add DEBUGPY=1 to debug)
uv run pytest                                # unit + integration
uv run pytest --cov=app --cov-report=term-missing
uv run playwright install chromium           # once
uv run pytest tests/e2e/test_smoke.py -v --no-cov
uv run ruff check . && uv run pyright        # lint + types
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt   # retrieval regression battery (phases 119/162)
uv run python -m scripts.api_payload_audit                      # endpoint byte audit vs the 256 KB budget (phase 167; app must be running)
```