brain-of-reese/AGENTS.md

AGENTS.md — Operating Rules for All Agents on This Repo

1. Always read .agents/PLAN.md first. It contains the architecture, the LOCKED DECISIONS, the UI/UX standards, the data model, and the roadmap. Do not design anything that contradicts it.

2. Follow the phased execution protocol in .agents/phases/. Work through .agents/phases/todo/ in numeric order. When a phase is complete, move its directory to .agents/phases/complete/.

3. Strictly adhere to the LOCKED DECISIONS (PLAN §2). If you believe a LOCKED decision is wrong, stop and flag it — do not silently deviate.

4. "One Story, One Phase": each user story in .agents/user_stories/ corresponds to a distinct execution phase, and every phase — story-driven or owner-directed — includes its own dedicated Playwright E2E test suite (one test file for the phase, run in isolation).

5. "UI Structure Check": before finalizing any UI component, verify that it follows the layout principles in .agents/PLAN.md §7 (proper container usage, the 72rem centered reading column — every page rides the 72rem container at every viewport; phase 100 supersedes §7.1's 46rem chat column, full-width tables on the catalogue pages — no skinny wasted-space lists; the catalogue tables (RAG, Sources, History) are bounded scroll panels — max-height: min(70vh, 64rem) — with a sticky header row; a clipped action label shows an ellipsis hint (.cell-clip) with the full text kept reachable via the control's aria-label (phase 156); an in-flight turn never paints an empty answer bubble — the box is created empty as the reasoning anchor and .msg .bubble:empty { display: none } hides it until the first answer token (phase 166); an answer's inline [text](url), ![alt](src) and ####+ heading render through the ONE shared renderer into an allowlist-gated a.md-link (underlined, never a javascript:/data: element), a container-capped img.md-img (max-width: 100% — no horizontal overflow at 375px) and an h5/h6 sized into the existing type scale — never printed reference text (phase 180); an SVG document paints as a graphic — the viewer's image block adds .doc-image-img--vector (width: 100%, object-fit: contain) when format is svg so a viewBox-only drawing spans the .doc-image card instead of the 300×150 default box, while the chat figure and RAG-table thumbnail keep their fixed caps and the bytes stay an <img> src (never inlined) (phase 181); the document viewer's page strip is CLASS-scoped — the primary class shows the strip in place of the text, the additive class never hides the text: Text/Pages toggle group (aria-pressed, Text default) with the strip behind it (phase 163); a lazily-drilled KB-tree level keeps the level the user is ON rendered while its node fetch runs, and its ONE in-flight row (role="status") / alert + Retry row (role="alert", a real <button>) sits inside that same bounded scroll panel — and the stat cards read the server's totals, never a walk of a depth-limited tree (phase 167); an answer's code block is EITHER verbatim-and-badged OR plainly the model's — never a silently-substituted hybrid: the ```bor locator fence renders through the ONE shared renderer as .verbatim[data-bor-locator] with the model's body DROPPED (renderMarkdown(md, { verbatim: true }), off by default so every other surface is byte-identical), styled from palette variables only with a min(40rem, 70vh) scroll body, a sticky source/path caption and a user-select: none line-number gutter, while the plain .code-block wrapper carries a Copy button and no badge (phase 182); those placeholders hydrate against the frame's answer_text through the ONE panel client (frontend/assets/verbatim.js — a classic script loaded AFTER markdown.js on the chat AND shared pages, no second locator grammar, a plain string compare against the server's canonical locator, the caption's numbers always the SERVER's extent, an unresolvable locator painting the pinned sentence I could not verify this against the knowledge base with no badge, no body and none of the model's bytes, no panel left at "Retrieving the exact text…" after a terminal frame (the error frame, the 300 s guard, Stop and the failure funnel all settle it), the Copy button's clipboard content equal to the server string exactly, and the shared page's panel Copy button + plain /document.html link the ONE owner-directed exception to that page's zero-controls rule) — phase 182 task 07), and the sync's peak-memory clause is TRAILING TEXT on the result line that already exists ( · peak <N> MB on the RAG page's #sync-progress counts line or "Sync cancelled", written by the ONE fmtSyncPeakClause off rss_peak_mb ?? rss_mb — no new control, no new live region, no new element, no clause in a running label or in the failed terminal's empty box, and NOTHING rendered when the pair is null — phase 187 task 03) and meets WCAG 2.1 AA basics (semantic landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the stream).

6. "No CDN Rule": all CSS, JS, fonts, and images must be served statically from within the FastAPI application. No <script src="https://…"> or <link href="https://…"> tags in HTML templates unless bundled locally. An integration test enforces this on every served page except /doc-edit.html (the parametrized no-CDN check in tests/integration/test_api.py — the shell, the viewer, the login and the shared page).

7. "Debugpy Check": any new entrypoint must keep the conditional import — debugpy is imported only when DEBUGPY=1 (see app/core/debugging.py); default off, never imported otherwise.

8. Git protocol: Git is mandatory. One atomic, professional commit per completed phase (Conventional Commits, e.g. feat(rag): stream RAG answers over SSE with source citations). Always append --no-gpg-sign (the repo also sets commit.gpgsign=false). .agents/ is tracked and committed with the phase — only runtime artifacts (.agents/phase-sessions/, .agents/pipeline.log) are gitignored.

9. Test gates are non-negotiable: a phase is complete only when unit + integration tests pass, app/ coverage is >90% (uv run pytest --cov=app --cov-report=term-missing), and the phase's Playwright E2E file passes in isolation (uv run pytest tests/e2e/test_<story>.py -v --no-cov).

10. Amply log, visibly feed back: server-side, log the required per-turn line (PLAN §9 — phase 182 appends verbatim_blocks=N after scaffold_stripped=N: the verbatim display blocks actually SENT on the done frame, 0 on every turn without them, deflected included) and the per-response byte-budget line (app/core/response_budget.py — one api method=… path=… status=… bytes=… enc=… duration_ms=… line per /api/* response, INFO at/over BOR_LOG_LARGE_RESPONSE_BYTES, read it with grep ' :: api ', audit it with uv run python -m scripts.api_payload_audit, phase 167) and the sync's memory fields (phase 187 — app/core/memory.py reads PROCESS RSS, never the host's cgroup figure, and never raises: a None renders ? through the ONE fmt_mb helper. rss_mb= is appended to sync: started … and to the importer's import: summary …, and rss_mb= + rss_peak_mb= to all three terminals (sync: done …, sync: cancelled by user …, and the sanitized one-line sync: failed state=failed … error=… that sits beside — never in place of — the logger.exception traceback). Append-only: no pre-existing field renamed, reordered or re-typed, so every older grep still matches; read them with grep ' :: ' server.log | grep ' rss_peak_mb='. Phase 187 task 03 puts the SAME pair on the status: SyncStatus.rss_mb / rss_peak_mb, written by the ONE _sample_rss — forced at the run start, at every phase transition and in every terminal (which keep the FINAL pair beside the final counts, the phase-64 convention, and log exactly what the payload reports) — and throttled to one read per _RSS_SAMPLE_INTERVAL_S (5 s) from the per-file progress hook, so a 27k-file walk costs ~600 /proc reads and not 27k (LOCKED A4). GET /api/sync/status reports both ADDITIVELY as rss_mb / rss_peak_mb (int | null; null is "no reading", never 0) and the failed funnel samples too, because the run that died is the run whose memory matters. That ONE sampler also owns the pair's invariant: the kernel refreshes ru_maxrss lazily, so a raw peak can sit BELOW the raw VmRSS (measured 164 vs 162) and _sample_rss lifts it to the current reading — every surface (both lines, the payload, the page's clause) therefore satisfies peak >= current, never a peak below the run's own footprint, and an unknowable peak is never fabricated from the current reading); UI-side, honor the "never stale" feedback contract (PLAN §7.4). Phase 188 adds the retrieval tier's ONE operator-visible line: when BOR_RETRIEVAL_EF_SEARCH sits below the A7 candidate count _vector_candidates logs retrieval: ef_search=%d raised to %d (A7 candidate count) at WARNING and runs the search at the candidate count anyway — the clamp is never silent, and the healthy (ef_search >= candidate count) turn logs nothing. The value itself reaches Postgres as SET LOCAL hnsw.ef_search inside the turn's own transaction (LOCKED A4: no global GUC, nothing survives the request or a pool recycle). Task 04's live gate (the 1,000-document live-replica KB, the live embedding endpoint, question vectors PINNED so the access path is the one variable — benchmarks/phase188_live_gate/) found no A8, top-5 or top_score difference at any depth and shipped retrieval_ef_search = 400: at the candidate count the graph recovered the exact top-100 for 0.9814 of rows on average (0.94 worst), at 400 for all of them, and NO threshold (relevance_threshold, lexical_support_floor, name_hit_bonus, hybrid_top_k, hybrid_vector_candidates) moved to get there (LOCKED A3). Two things the operator needs from the same run: building the index costs ~161 s per 417k chunks and ~1.6 GB (3.9 KB/row at m = 16, 768-dim), and maintenance_work_mem = '512MB' is NOT applicable in this container — /dev/shm is 63 MB, so a raise above that dies with could not resize shared memory segment … No space left on device and leaves an INVALID index for the migration's self-heal to drop and rebuild (observed live); the container's 64 MB default builds it fine.

11. Never block the event loop: anything blocking (a subprocess, synchronous file/network I/O, a heavy walk) reached from an async handler or a spawned background coroutine goes through asyncio.to_thread (or create_subprocess_exec). The sync's git calls do (phase 179: a long git clone used to park the single uvicorn loop and freeze the whole HTTP surface, the 2 s GET /api/sync/status poll included — pinned by tests/integration/test_sync_git_freeze.py with a 3 s stub and by tests/e2e/test_sync_git_freeze.py with a real ~9 s file:// clone). A cooperative cancel unblocks the run at the to_thread await; the worker thread itself runs to completion (threads are not interruptible) while the loop keeps serving — pinned by the same E2E module's cancel-during-clone test. Since phase 186 scripts/git_sync.py has two git spawn sites — the buffered run_git and the streaming stream_git_lines (the commit-date walk yields git log line by line instead of holding the whole history in RAM) — and both are off-loop via asyncio.to_thread; the subprocess. single-site intent survives as "one per shape, both owned there", enforced by tests/unit/test_git_sync.py::test_only_two_git_spawn_sites, and the owner-visible date contract through the stream (last-commit dates, the mtime fallback, the D1 manual lock) by tests/e2e/test_git_log_stream_dates.py.

12. Never hydrate what you delete: Document.chunks is passive_deletes=True (phase 183), so deleting a document removes its chunk rows in the database (chunks_document_id_fkey is ON DELETE CASCADE, alembic 0001) and a mass prune costs O(documents). The importer's prune and the source-removal endpoint both rely on it. Do not remove the keyword, and do not add a delete path that SELECTs chunks (or a Chunk.embedding) on its way to deleting them — one hydrated 768-dim embedding is ~24.8 KiB of Python, which is how the 2026-10-15 sync's 3,008-document prune reached 4 GB and locked up the VM. Pinned by tests/integration/test_prune_cascade_statements.py (statement count, zero orphan chunk rows, and a live-FK drift guard that fails the suite if the cascade clause ever disappears — because then passive_deletes=True would silently orphan rows). The doc.chunks = [...] re-embed path is the deliberate, documented exception (collection replacement legitimately needs the loaded collection; changed files only).

13. Release the sync session at the proven-safe points only: an import run holds one session and expire_on_commit=False (app/db.py) means a commit releases nothing, so _unchanged_doc's file_hash stamp keeps every walked Document — content included — resident in the session's own strong dirty set (not collectable): memory grew with the KB. The cadence is therefore session.expunge_all() every BOR_SYNC_SESSION_RESET_FILES files at the TOP of the per-file loop and once per source before the two post-passes (phase 184; 0 = never, the pre-phase rollback). expunge_all DISCARDS unflushed changes, so never move it into _index_file, _unchanged_doc, _store_summary or a post-pass — their locals are live across statements — and never add a reset before a file's per-file commit. Pinned by tests/integration/test_import_session_retention.py (bounded with the cadence on, unbounded with it 0, byte-identical summaries and DB) and through the UI by tests/e2e/test_sync_session_reset.py (two module apps, one per cadence: every document exactly once, the all-unchanged re-sync, and the 0-cadence run byte-identical to the 3-cadence one).

14. The prune reads six columns and commits in batches: _prune selects its candidates through load_only(id, source, path, is_image, image_path, page_count) order_by(Document.id) — the exact set the phase-122 images-off guard, the phase-126 parent-PDF rule and the image/page copy cleanup read, and NOT content (252 MB of markdown per full KB walk) — and commits every BOR_SYNC_PRUNE_BATCH deletes (0 = the pre-phase single commit; phase 184). Never read another Document attribute inside that loop (a deferred column costs one lazy SELECT per document and unbounds the walk — no raiseload paper-over, add the column instead), and never swap the per-document session.delete(doc) for a bulk delete() (the guards, changed_paths and the log line depend on the per-document shape). A non-zero batch deliberately trades the all-or-nothing prune for a prefix (LOCKED A2): seen is recomputed from disk, so the next sync converges. Pinned by tests/integration/test_import_session_retention.py (the captured SELECT's column list, exactly one document-wide SELECT per prune (a second one is a per-row lazy load), the commit counts at 10 and 0, both guards on light rows) and tests/e2e/test_sync_session_reset.py (the five-document prune at batch 2 through the RAG page (/sources.html): both folders cleared, the survivors' chunks intact, zero orphan chunk rows).

15. The unchanged fast path reads metadata, never markdown: the four row lookups that share it (_index_file, _index_pdf_file, _index_one_embedded_image, _index_image_file) all go through the ONE statement _doc_row_stmt (app/rag/importer.py), which defers Document.content and NOTHING ELSE (phase 185, LOCKED A1) — the row load is the only place the fast path gets its row, and hydrating content for every inspected file is what put ~252.6 MB of markdown in front of a run that re-indexed 217 of ~27k files. Do not add a fifth whole-entity select(Document).where(Document.source == …, Document.path == …), do not swap the defer for a load_only allowlist (a future column would silently become lazy and resurface as an N+1), and do not add raiseload/noload (LOCKED A2): _unchanged_doc's phase-118 summary-NULL backfill genuinely reads doc.content and must pay one per-row, one-time SELECT instead of raising. Pinned by tests/integration/test_import_fast_path_columns.py (the captured fast-path SELECT names no documents.content, exactly one SELECT per unchanged file with a summary, exactly one PK follow-up for the backfill), tests/unit/test_importer_defer_content.py (content is the only deferred column, the four loads share the helper, no raiseload/noload in the importer) and tests/e2e/test_fast_path_deferred_content.py (the owner-visible parity: the summary-NULL backfill survives the deferral with exactly one is_summary chunk, an unchanged doc's viewer shows the full stored text, a changed file re-indexes and the fast path still fires after).

16. The cosine tier is indexed — keep it indexed, and keep 0029 the ONE definition of that index: ix_chunks_embedding_hnsw (HNSW over chunks.embedding, vector_cosine_ops, m = 16, ef_construction = 64) exists because every turn's ORDER BY embedding <=> :vec LIMIT 100 full-scanned ~3 GB on the interactive path (phase 188). Build and drop it ONLY through migration 0029 (CREATE/DROP INDEX CONCURRENTLY inside op.get_context().autocommit_block(), which pre-drops an INVALID remnant and raises if the build does not validate) — never an inline CREATE INDEX in app code, a test, or a runbook step; never a second vector index, a halfvec cast, or a rewritten ORDER BY (the <=> operator is what the graph serves, LOCKED A3/A6). A test that needs the index OFF moves the SCHEMA (command.downgrade(cfg, "0028") / command.upgrade(cfg, "0029")) after rolling back its own session — an open snapshot parks a CONCURRENTLY build indefinitely — and asserts the catalog state it claims (pg_index.indisvalid, pg_get_indexdef). hnsw.ef_search is set with SET LOCAL inside the turn's own transaction and the int()-coerced setting is the only source (no global SET, no pool-wide state, nothing survives a recycle); the shipped default is 400 because that is the depth the live gate measured exact-top-100 parity at, so "tuning" it back to the candidate count is a recall change, and closing a recall gap by moving an honesty threshold is forbidden (LOCKED A3). The ef_search a turn gets is max(retrieval_ef_search, hybrid_vector_candidates) with a WARNING when clamped. Pinned by tests/integration/test_chunks_hnsw_index.py (DDL round trip + INVALID self-heal + drift guard), tests/integration/test_chunks_hnsw_recall.py (the tier's captured statement names the index, SET LOCAL reaches the connection, recall at the shipped depth with the index paths forced — enable_seqscan, enable_indexonlyscan AND enable_bitmapscan off, because a bitmap scan through ix_chunks_document_id otherwise disguises a full read and flips the plan on a shared dev DB — and the unforced plan asserted only when the relation holds nothing but the module's own corpus), and tests/e2e/test_vector_index_recall.py (the owner-visible arm: the SAME cited document with the index dropped and restored, the index VALID after a real importer run, the near-miss still deflecting in both arms, and the clamp read off the app's own captured stdout at =150 / =50). That suite's sentinels are calibrated against tests/fixtures/hnswkb/ — an 11-note tracked KB whose BYTES own the quoted cosine/lexical margins, so editing a note there moves the numbers in the module docstring and the pins together.

17. A source label is an alias, never an identity (phase 191): the source_labels row (alembic 0030, PK = the DERIVED source name exactly as Document.source carries it) holds the owner's display_name / label / tags, and only app/api/source_labels.py writes it — GET /api/source-labels (the catalogue: the effective_sources rows of BOTH kinds — or the BOR_GIT_SOURCES names while the table is empty — plus every indexed document source no row derives, de-duplicated and sorted, ONE all_labels read for the whole list), PUT (partial update: omitted keeps, explicit null clears, tags: [] clears) and DELETE (the reset); require_admin router-wide, and every name resolution goes through app.rag.source_removal.resolve_source_name — never re-derive the basename here or the row keys nothing. A PUT for a name that is neither a registry row's derived name nor an indexed document source is 404 (no orphan label rows) and a validation failure writes NOTHING. NEVER rename, resolve or join on the display name: Document.source, the source/path read/grep/show_text locators, the citations, the KB tree and the verbatim caption stay on the derived name, so a rename cannot invalidate a saved chat, a shared page or a locator — and the name reaches the model only through the prompt/retrieval layers (tasks 04/05), never by replacing the identity in a chip, a table or a caption. Pinned by tests/integration/test_source_labels_api.py (the 403 gate for anonymous AND token users, the git+local+upload union sorted and de-duplicated at three SELECTs with no documents.content read, the partial-update shape, every cap as the framework's 422 with the row byte-identical, both 404 arms, the ValueError → 422 funnel, and the "a rename writes nothing but the label row" arm). The Sources-page editor (task 03) is the owner-visible half of the SAME contract and must stay that way: its row list comes from GET /api/source-labels (the catalogue union — never a re-derived basename, never a join on the display name), the row's identity cell is a read-only <code> of the DERIVED name (an input there WOULD be the alias-contract bug), Save/Reset PUT/DELETE the per-source path and re-read the ROW from that response (no second GET), the save state is scoped PER ROW (saveInFlight — one request per row, the finally always settles it, the 404 note names the arm, a collapse discards the draft) — and a FAILED save's note SURVIVES that settle: updateLabelRowUI returns before the dirty hint whenever the line carries .is-error, so "That source no longer exists — reload the page." cannot be painted over by "unsaved changes" in the tick that wrote it (the defect the task-06 story suite walked into); only the next edit (clearLabelError) or a collapse clears a note, never the settle path), and the Tags token input's chips reuse the phase-89 tag rule (.source-label-tag sits in the .git-source-hidden-count rule group — no bespoke tag-input CSS). Pinned by tests/unit/test_source_labels_editor_ui.py and, through the real UI over one git + one local + one uploaded source, by tests/e2e/test_source_name_labels.py (task 06: the editor saves and re-populates all three kinds, a question that USES the name moves that source's document into the seeded <documents> tier with name/label on the wire while the same row with a name-free question moves nothing, the vanished-source 404 note is visible, Reset reverts the tier, and the alias never appears in a locator, a nearby-doc link, the KB tree or the viewer); the chat source chips, the RAG source tree and the verbatim caption keep the derived name untouched. The selection boost (task 04) is the retrieval half of the SAME contract: ONE all_labels read per turn at the retrieval boundary (app/api/chat.py, after retrieve, before the first selection call) flags RetrievedChunk.source_name_hit on the chunks whose document.source is a D2 hit (the pure question_name_hits rule applied per view — a loop over the few dozen chunks, never a per-chunk query; that ONE result is also what task 05's prompt surfaces consume, so a turn issues one label query and the standalone question_name_hit_sources wrapper never runs in one), and _selection_order adds source_name_hit_bonus (default 0.005; 0 = the byte-identical kill switch; a negative value refuses startup) as a SECOND, independent bonus — once per DOCUMENT, at most two bonuses stacked with the phase-119 name_hit_bonus (bounded 0.010 on the fused scale), SELECTION layer only (score/cosine/fts_hit/ query_log.top_score and the per-turn log line untouched). Pinned by tests/unit/test_retriever.py (the kill-switch byte-identity, the bounded two-bonus application, the fuse OR-merge, the three selection helpers pulling the knob from settings). The prompt surfacing (task 05) is the model-facing half of the SAME read: build_high_prompt(names=…) gives the suggested <document> block the ADDITIVE name="…" (immediately after source="…") and label="…" attributes — the order is locked (source → name? → label? → path → title → date), the two values escape through the ONE _attr helper, and the identity attributes keep their pre-phase unescaped bytes; regenerate_overview derives the KB-overview INPUT lines' name map from its own ONE all_labels read (build_overview_prompt(names=…) swaps the raw source name for the display name in the lite model's input ONLY — the stored outline and its deterministic Sources: … tail stay derived name). An unset field adds NOTHING (never name=""), tags never render, the LOW prompt never carries a name, and no map / no entry is byte-identical to pre-phase on BOTH builders (the kill pins in tests/unit/test_prompts.py and tests/unit/test_overview.py, which also marker-pin the one-read-per-surface contract, as tests/unit/test_chat_gate.py does on the chat call site).

Quick reference

podman compose up -d db                      # start Postgres 17 + pgvector
cp .env.example .env                         # once; then edit
uv run alembic upgrade head                  # apply migrations
uv run uvicorn app.main:app --reload         # dev server (add DEBUGPY=1 to debug)
uv run pytest                                # unit + integration
uv run pytest --cov=app --cov-report=term-missing
uv run playwright install chromium           # once
uv run pytest tests/e2e/test_smoke.py -v --no-cov
uv run ruff check . && uv run pyright        # lint + types
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt   # retrieval regression battery (phases 119/162)
uv run python -m scripts.api_payload_audit                      # endpoint byte audit vs the 256 KB budget (phase 167; app must be running)