brain-of-reese/AGENTS.md

AGENTS.md — Operating Rules for All Agents on This Repo

1. Always read .agents/PLAN.md first. It contains the architecture, the LOCKED DECISIONS, the UI/UX standards, the data model, and the roadmap. Do not design anything that contradicts it.
2. Follow the phased execution protocol in .agents/phases/. Work through .agents/phases/todo/ in numeric order. When a phase is complete, move its directory to .agents/phases/complete/.
3. Strictly adhere to the LOCKED DECISIONS (PLAN §2). If you believe a LOCKED decision is wrong, stop and flag it — do not silently deviate.
4. "One Story, One Phase": each user story in .agents/user_stories/ corresponds to a distinct execution phase, and every phase — story-driven or owner-directed — includes its own dedicated Playwright E2E test suite (one test file for the phase, run in isolation).
5. "UI Structure Check": before finalizing any UI component, verify that it follows the layout principles in .agents/PLAN.md §7 (proper container usage, the 72rem centered reading column — every page rides the 72rem container at every viewport; phase 100 supersedes §7.1's 46rem chat column, full-width tables on the catalogue pages — no skinny wasted-space lists; the catalogue tables (RAG, Sources, History) are bounded scroll panels — max-height: min(70vh, 64rem) — with a sticky header row; a clipped action label shows an ellipsis hint (.cell-clip) with the full text kept reachable via the control's aria-label (phase 156); an in-flight turn never paints an empty answer bubble — the box is created empty as the reasoning anchor and .msg .bubble:empty { display: none } hides it until the first answer token (phase 166); an answer's inline [text](url), ![alt](src) and ####+ heading render through the ONE shared renderer into an allowlist-gated a.md-link (underlined, never a javascript:/data: element), a container-capped img.md-img (max-width: 100% — no horizontal overflow at 375px) and an h5/h6 sized into the existing type scale — never printed reference text (phase 180); an SVG document paints as a graphic — the viewer's image block adds .doc-image-img--vector (width: 100%, object-fit: contain) when format is svg so a viewBox-only drawing spans the .doc-image card instead of the 300×150 default box, while the chat figure and RAG-table thumbnail keep their fixed caps and the bytes stay an <img> src (never inlined) (phase 181); the document viewer's page strip is CLASS-scoped — the primary class shows the strip in place of the text, the additive class never hides the text: Text/Pages toggle group (aria-pressed, Text default) with the strip behind it (phase 163); a lazily-drilled KB-tree level keeps the level the user is ON rendered while its node fetch runs, and its ONE in-flight row (role="status") / alert + Retry row (role="alert", a real <button>) sits inside that same bounded scroll panel — and the stat cards read the server's totals, never a walk of a depth-limited tree (phase 167); an answer's code block is EITHER verbatim-and-badged OR plainly the model's — never a silently-substituted hybrid: the ```bor locator fence renders through the ONE shared renderer as .verbatim[data-bor-locator] with the model's body DROPPED (renderMarkdown(md, { verbatim: true }), off by default so every other surface is byte-identical), styled from palette variables only with a min(40rem, 70vh) scroll body, a sticky source/path caption and a user-select: none line-number gutter, while the plain .code-block wrapper carries a Copy button and no badge (phase 182); those placeholders hydrate against the frame's answer_text through the ONE panel client (frontend/assets/verbatim.js — a classic script loaded AFTER markdown.js on the chat AND shared pages, no second locator grammar, a plain string compare against the server's canonical locator, the caption's numbers always the SERVER's extent, an unresolvable locator painting the pinned sentence I could not verify this against the knowledge base with no badge, no body and none of the model's bytes, no panel left at "Retrieving the exact text…" after a terminal frame (the error frame, the 300 s guard, Stop and the failure funnel all settle it), the Copy button's clipboard content equal to the server string exactly, and the shared page's panel Copy button + plain /document.html link the ONE owner-directed exception to that page's zero-controls rule) — phase 182 task 07) and meets WCAG 2.1 AA basics (semantic landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the stream).
6. "No CDN Rule": all CSS, JS, fonts, and images must be served statically from within the FastAPI application. No <script src="https://…"> or <link href="https://…"> tags in HTML templates unless bundled locally. An integration test enforces this on the index page.
7. "Debugpy Check": any new entrypoint must keep the conditional import — debugpy is imported only when DEBUGPY=1 (see app/core/debugging.py); default off, never imported otherwise.
8. Git protocol: Git is mandatory. One atomic, professional commit per completed phase (Conventional Commits, e.g. feat(rag): stream RAG answers over SSE with source citations). Always append --no-gpg-sign (the repo also sets commit.gpgsign=false). .agents/ is tracked and committed with the phase — only runtime artifacts (.agents/phase-sessions/, .agents/pipeline.log) are gitignored.
9. Test gates are non-negotiable: a phase is complete only when unit + integration tests pass, app/ coverage is >90% (uv run pytest --cov=app --cov-report=term-missing), and the phase's Playwright E2E file passes in isolation (uv run pytest tests/e2e/test_<story>.py -v --no-cov).
10. Amply log, visibly feed back: server-side, log the required per-turn line (PLAN §9 — phase 182 appends verbatim_blocks=N after scaffold_stripped=N: the verbatim display blocks actually SENT on the done frame, 0 on every turn without them, deflected included) and the per-response byte-budget line (app/core/response_budget.py — one api method=… path=… status=… bytes=… enc=… duration_ms=… line per /api/* response, INFO at/over BOR_LOG_LARGE_RESPONSE_BYTES, read it with grep ' :: api ', audit it with uv run python -m scripts.api_payload_audit, phase 167); UI-side, honor the "never stale" feedback contract (PLAN §7.4).
11. Never block the event loop: anything blocking (a subprocess, synchronous file/network I/O, a heavy walk) reached from an async handler or a spawned background coroutine goes through asyncio.to_thread (or create_subprocess_exec). The sync's git calls do (phase 179: a long git clone used to park the single uvicorn loop and freeze the whole HTTP surface, the 2 s GET /api/sync/status poll included — pinned by tests/integration/test_sync_git_freeze.py with a 3 s stub and by tests/e2e/test_sync_git_freeze.py with a real ~9 s file:// clone). A cooperative cancel unblocks the run at the to_thread await; the worker thread itself runs to completion (threads are not interruptible) while the loop keeps serving — pinned by the same E2E module's cancel-during-clone test. Since phase 186 scripts/git_sync.py has two git spawn sites — the buffered run_git and the streaming stream_git_lines (the commit-date walk yields git log line by line instead of holding the whole history in RAM) — and both are off-loop via asyncio.to_thread; the subprocess. single-site intent survives as "one per shape, both owned there", enforced by tests/unit/test_git_sync.py::test_only_two_git_spawn_sites, and the owner-visible date contract through the stream (last-commit dates, the mtime fallback, the D1 manual lock) by tests/e2e/test_git_log_stream_dates.py.
12. Never hydrate what you delete: Document.chunks is passive_deletes=True (phase 183), so deleting a document removes its chunk rows in the database (chunks_document_id_fkey is ON DELETE CASCADE, alembic 0001) and a mass prune costs O(documents). The importer's prune and the source-removal endpoint both rely on it. Do not remove the keyword, and do not add a delete path that SELECTs chunks (or a Chunk.embedding) on its way to deleting them — one hydrated 768-dim embedding is ~24.8 KiB of Python, which is how the 2026-10-15 sync's 3,008-document prune reached 4 GB and locked up the VM. Pinned by tests/integration/test_prune_cascade_statements.py (statement count, zero orphan chunk rows, and a live-FK drift guard that fails the suite if the cascade clause ever disappears — because then passive_deletes=True would silently orphan rows). The doc.chunks = [...] re-embed path is the deliberate, documented exception (collection replacement legitimately needs the loaded collection; changed files only).
13. Release the sync session at the proven-safe points only: an import run holds one session and expire_on_commit=False (app/db.py) means a commit releases nothing, so _unchanged_doc's file_hash stamp keeps every walked Document — content included — resident in the session's own strong dirty set (not collectable): memory grew with the KB. The cadence is therefore session.expunge_all() every BOR_SYNC_SESSION_RESET_FILES files at the TOP of the per-file loop and once per source before the two post-passes (phase 184; 0 = never, the pre-phase rollback). expunge_all DISCARDS unflushed changes, so never move it into _index_file, _unchanged_doc, _store_summary or a post-pass — their locals are live across statements — and never add a reset before a file's per-file commit. Pinned by tests/integration/test_import_session_retention.py (bounded with the cadence on, unbounded with it 0, byte-identical summaries and DB) and through the UI by tests/e2e/test_sync_session_reset.py (two module apps, one per cadence: every document exactly once, the all-unchanged re-sync, and the 0-cadence run byte-identical to the 3-cadence one).
14. The prune reads six columns and commits in batches: _prune selects its candidates through load_only(id, source, path, is_image, image_path, page_count) order_by(Document.id) — the exact set the phase-122 images-off guard, the phase-126 parent-PDF rule and the image/page copy cleanup read, and NOT content (252 MB of markdown per full KB walk) — and commits every BOR_SYNC_PRUNE_BATCH deletes (0 = the pre-phase single commit; phase 184). Never read another Document attribute inside that loop (a deferred column costs one lazy SELECT per document and unbounds the walk — no raiseload paper-over, add the column instead), and never swap the per-document session.delete(doc) for a bulk delete() (the guards, changed_paths and the log line depend on the per-document shape). A non-zero batch deliberately trades the all-or-nothing prune for a prefix (LOCKED A2): seen is recomputed from disk, so the next sync converges. Pinned by tests/integration/test_import_session_retention.py (the captured SELECT's column list, exactly one document-wide SELECT per prune (a second one is a per-row lazy load), the commit counts at 10 and 0, both guards on light rows) and tests/e2e/test_sync_session_reset.py (the five-document prune at batch 2 through the RAG page (/sources.html): both folders cleared, the survivors' chunks intact, zero orphan chunk rows).
15. The unchanged fast path reads metadata, never markdown: the four row lookups that share it (_index_file, _index_pdf_file, _index_one_embedded_image, _index_image_file) all go through the ONE statement _doc_row_stmt (app/rag/importer.py), which defers Document.content and NOTHING ELSE (phase 185, LOCKED A1) — the row load is the only place the fast path gets its row, and hydrating content for every inspected file is what put ~252.6 MB of markdown in front of a run that re-indexed 217 of ~27k files. Do not add a fifth whole-entity select(Document).where(Document.source == …, Document.path == …), do not swap the defer for a load_only allowlist (a future column would silently become lazy and resurface as an N+1), and do not add raiseload/noload (LOCKED A2): _unchanged_doc's phase-118 summary-NULL backfill genuinely reads doc.content and must pay one per-row, one-time SELECT instead of raising. Pinned by tests/integration/test_import_fast_path_columns.py (the captured fast-path SELECT names no documents.content, exactly one SELECT per unchanged file with a summary, exactly one PK follow-up for the backfill), tests/unit/test_importer_defer_content.py (content is the only deferred column, the four loads share the helper, no raiseload/noload in the importer) and tests/e2e/test_fast_path_deferred_content.py (the owner-visible parity: the summary-NULL backfill survives the deferral with exactly one is_summary chunk, an unchanged doc's viewer shows the full stored text, a changed file re-indexes and the fast path still fires after).

Quick reference

podman compose up -d db                      # start Postgres 17 + pgvector
cp .env.example .env                         # once; then edit
uv run alembic upgrade head                  # apply migrations
uv run uvicorn app.main:app --reload         # dev server (add DEBUGPY=1 to debug)
uv run pytest                                # unit + integration
uv run pytest --cov=app --cov-report=term-missing
uv run playwright install chromium           # once
uv run pytest tests/e2e/test_smoke.py -v --no-cov
uv run ruff check . && uv run pyright        # lint + types
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt   # retrieval regression battery (phases 119/162)
uv run python -m scripts.api_payload_audit                      # endpoint byte audit vs the 256 KB budget (phase 167; app must be running)