chore(agent): phase roadmap from TODO.md, 4 phases
Protocol B append (phases 146-149) for the four TODO items: the search tool line missing the grep's document scope (146), sync-only- what-changed via a raw-file byte hash checked before extraction and vision (147), pictorial PDFs showing their rendered pages in the sources viewer instead of page descriptions (148), and the sync detail moved to a text box under the button with the button as a spinner (149). TODO.md cleared (items now live in .agents/phases/ todo/). LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
# Phase 147 — sync only what changed: the source file's byte hash is checked BEFORE extraction and vision
|
||||
|
||||
**Source:** `TODO.md` L4 — "Local sources, like PDFs, are synced every time even if they haven't changed. This results in really long sync times for PDFs that are mostly pictoral. Only sync documents if they've changed since the last sync (hash the document contents to check)."
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/rag/importer.py` — `import_sources` (L560: the per-source walk; the `progress`/`on_chunk_progress` hooks fire per file before indexing — unchanged by this phase), `_index_file` (L879: the single upsert entry — dispatches image → `_index_image_file` (L2642), `.pdf` → `_index_pdf_file` (L1181), text fallback), and the three per-branch unchanged blocks that today run only AFTER the expensive work: text (L~928: digest over the read text — cheap, the read itself is the cost), image (L~2660: digest over `read_bytes()` — the vision description runs only on a digest MISS, already fine), PDF (L~1303: digest over the FINAL constructed content — reached only after `extract_pdf_text` (L162 of `app/rag/pdf.py`) ALWAYS runs and, for garbage layers, `_fallback_page_screenshots` (every page rendered + vision-described) or, for sparse layers with `pdf_pictorial` on, `_pictorial_pdf_pass` (every candidate page rendered + vision-described) — so an UNCHANGED pictorial PDF re-extracts and re-burns vision on EVERY sync; a non-deterministic vision model additionally flips the final-content digest and forces a full re-chunk/re-embed/re-summary each run — the owner's "synced every time even if they haven't changed"). `Document.content_hash` (app/models.py L112) is the FINAL-CONTENT digest (exposed via `app/rag/retriever.py` L203/426/555/611 — the dedupe/caching identity) and stays untouched; the new `file_hash` is the RAW SOURCE-FILE-BYTES digest, compared first, used for change detection only. `alembic/versions/` — the latest migration is `0024_git_source_branch.py`. The unchanged-block semantics that must survive the fast path (all three branches, identical shape): `summary.unchanged += 1` + log; the phase-118 (A2) summary-NULL backfill; the phase-106 (D1) `created_at_manual` early-return; the phase-106 (D4) date refresh (`normalize_doc_date` of the raw_date — mtime fallback — may go OLDER, counts `summary.dates_updated`, commits).
|
||||
**Locked (owner delegation 2026-09-30, chat — "pick the best choices"):** change detection = sha256 of the RAW FILE BYTES (the owner's "hash the document contents"), computed ONCE at the top of `_index_file` — before the image/text/PDF dispatch — and compared against a new `documents.file_hash` column (migration 0025, nullable). A match takes a shared unchanged fast path: NO file read (beyond the streamed hash), NO PDF extraction, NO vision call, NO embed, NO summary call — only the preserved bookkeeping (summary-NULL backfill from the STORED `doc.content`, manual-date early-return, date refresh). Pre-phase rows (`file_hash` NULL) fall through to the branches' existing logic exactly as today and get their hash STAMPED on that first post-upgrade sync (one full pass, then fast forever). `content_hash` and everything built on it (retriever exposure, chunk identity) are untouched.
|
||||
|
||||
## Objective
|
||||
An unchanged source file costs one streamed sha256 and a row lookup per sync — zero extraction, zero vision, zero embedding, zero summary LLM — across all three document classes (text, image, pictorial PDF), while every preserved unchanged-path behavior (summary backfill, manual-date lock, date refresh, counts) is byte-for-byte maintained.
|
||||
|
||||
## Dependencies
|
||||
- `146_search_tool_scope` (todo) — pipeline order only; no shared code (this phase touches the importer + one migration; 146 touches the tool-frame wire + the chat renderer).
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The column (task 01):** `alembic/versions/0025_documents_file_hash.py` — `documents.file_hash` `String(64)`, **nullable, no index** (it is compared in Python after the unique `(source, path)` lookup — unlike `content_hash`, it is never a query surface and never exposed to the retriever); downgrade drops it. `Document.file_hash: Mapped[str | None]` + docstring (the raw source-file-bytes sha256 — the sync change signal; DISTINCT from `content_hash` — the final indexed content's digest, which for a pictorial PDF includes vision text and is the retriever's identity).
|
||||
- **The fast path (task 02):**
|
||||
- `_file_hash(path: Path) -> str` — sha256 over the raw bytes, streamed in 1 MiB chunks (a multi-hundred-MB PDF never loads fully; the `hashlib` incremental pattern).
|
||||
- `_index_file` top (before the image/PDF/text dispatch): `file_digest = _file_hash(full_path)`; the row lookup (the `(source, path)` scalar select the branches already do — moved up); `if doc is not None and doc.file_hash == file_digest:` → `_unchanged_doc(...)` → **return**. The `raw_date` mtime fallback (the D2 convention: read before the unchanged early-return — the branches already do this; the fast path needs it for the date refresh) is computed in `_index_file` before the check when `raw_date is None`.
|
||||
- `_unchanged_doc(session, doc, *, source, rel, llm, summary, raw_date)` — the ONE shared implementation of the three near-identical unchanged blocks: `summary.unchanged += 1` + the `import: unchanged` log line (unchanged wording); the phase-118 (A2) backfill when `doc.summary is None` — `content=doc.content` (EXACTLY what the bytes produced — for text the stored content IS the file text; for an image the stored description; for a PDF the stored final content — so the backfill never re-reads/re-extracts/re-describes); the D1 `created_at_manual` early-return; the D4 date refresh (`target = normalize_doc_date(raw_date)`; on change: commit + `summary.dates_updated` + the `import: date-refreshed` log line — unchanged wording). The three branches' existing unchanged blocks are REPLACED by the shared helper (the per-branch callsite passes the same arguments; behavior is identical, code is one site).
|
||||
- **Hash stamping:** every branch writes `doc.file_hash = file_digest` on create AND update (all three — text, image, PDF — `file_digest` passed down from `_index_file`). The fast path makes the stamp redundant for the common case, but a bytes-changed/content-same edge (e.g. a PDF metadata edit with identical extracted text) lands in a branch whose content digest still matches — the branch's unchanged flow (now the shared helper) ALSO stamps `doc.file_hash = file_digest` and commits when it is dirty (the date-refresh commit may already have run — stamp BEFORE that block's final decision: set the attribute, and commit once at the end of the helper if dirty) so the NEXT sync takes the fast path. Pre-phase rows (`file_hash` NULL) take exactly this path on the first post-upgrade sync (full pipeline or the shared helper — whichever the branch's content digest decides) and are stamped either way.
|
||||
- The `progress`/`on_chunk_progress` hooks are untouched (the `progress` call fires per file before `_index_file` as today — an unchanged file was already "the current file" pre-phase; the fast path just makes it cheap). `summary.files` counts the walked file either way. Pruning, `limit`, the per-source post-index passes (`_enrich_pending_images`, `_backfill_image_contexts` — row-driven) are untouched.
|
||||
- **The E2E (task 03):** the `file://` fixture-repo module-app idiom (the `test_sync_button.py` recipe — real clone, mock LLM): run 1 (N added) → capture every document row's `indexed_at`/`content_hash` → run 2 → the settle line reads `0 added · 0 updated · N unchanged` and every row's `indexed_at`/`content_hash` is unchanged (the re-index would bump `indexed_at`) → commit a new file to the fixture repo → run 3 → only the new file is `added`.
|
||||
|
||||
## Tasks
|
||||
1. `01_file_hash_column.md` — migration 0025 + the model field.
|
||||
2. `02_unchanged_fast_path.md` — the streamed hash, the top-of-`_index_file` check, the shared unchanged helper, the stamping, the branch rewiring, the unit matrix.
|
||||
3. `03_e2e_unchanged_second_run.md` — the dedicated E2E: the idempotent second sync is provably free of re-indexing.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the fast-path matrix (text / image / garbage-PDF / pictorial-PDF unchanged → zero LLM calls; changed → full pipeline + stamp; NULL hash → pre-phase behavior + stamp; D1/D4 pins on the fast path; the streamed hash on a >1 MiB file).
|
||||
- Integration: the migration upgrade/downgrade round-trip (the house migration-test idiom).
|
||||
- E2E: one dedicated Playwright file, run in isolation, mock-only (real git clone).
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `documents.file_hash` exists (nullable String(64), no index) via migration 0025; the upgrade/downgrade round-trips on the test DB
|
||||
- [ ] an unchanged text file, image, garbage-layer PDF, and pictorial PDF each cost ZERO llm calls (embed + chat) and zero PDF extraction/vision on the second sync — `summary.unchanged` counts them; `indexed_at`/`content_hash` untouched
|
||||
- [ ] a changed file (any byte) re-indexes fully and stamps the new hash; a bytes-changed/content-same file stamps the hash without a content change
|
||||
- [ ] the preserved unchanged semantics are pinned: summary-NULL backfill, `created_at_manual` early-return, the date refresh (may go older, counts `dates_updated`, commits), the log-line wordings
|
||||
- [ ] `content_hash` and the retriever's exposure of it are byte-identical to pre-phase
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` green in isolation; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] no behavior change in completed phases (prune semantics, `limit`/`progress` contracts, the per-source post-index passes, the sync status/result wire)
|
||||
@@ -0,0 +1,24 @@
|
||||
# Task 01 — migration 0025: `documents.file_hash` (the raw source-file-bytes digest)
|
||||
|
||||
**Phase:** `147_sync_changed_only` · **Source:** `TODO.md:4` — "Local sources, like PDFs, are synced every time even if they haven't changed. This results in really long sync times for PDFs that are mostly pictoral. Only sync documents if they've changed since the last sync (hash the document contents to check)."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The `documents` table gains `file_hash` — the sha256 of the RAW SOURCE FILE's bytes — the column the task-02 fast path compares before any extraction or vision work. Existing rows stay NULL (= "unknown" — one full pipeline pass stamps them on the next sync; no backfill is possible in a migration — the migration cannot see the source files on disk).
|
||||
|
||||
## Work
|
||||
1. `alembic/versions/0025_documents_file_hash.py` — `revision = "0025"`, `down_revision = "0024"`:
|
||||
- `upgrade`: `op.add_column("documents", sa.Column("file_hash", sa.String(length=64), nullable=True))` — **no index** (the fast path compares it in Python after the unique `(source, path)` lookup; unlike `content_hash` — indexed because it is a query/exposure surface — `file_hash` is internal bookkeeping) and no server default (NULL is the meaningful pre-phase state).
|
||||
- `downgrade`: `op.drop_column("documents", "file_hash")`.
|
||||
2. `app/models.py` — `Document` (L102): add, directly below `content_hash` (L112), `file_hash: Mapped[str | None] = mapped_column(String(64), default=None)` with a docstring in the house style: the sha256 of the RAW SOURCE FILE's bytes at index time (phase 147) — the sync's change signal, compared at the top of the importer's `_index_file` BEFORE the image/text/PDF dispatch (an unchanged file costs one streamed hash + one lookup — no extraction, no vision, no embed). NULL for pre-phase rows and until the first post-upgrade sync stamps it. DISTINCT from `content_hash` — the FINAL indexed content's digest (a pictorial PDF's includes vision text; `content_hash` is exposed via `app/rag/retriever.py` and is the dedupe/caching identity — untouched).
|
||||
3. Tests (the house migration-test idiom — the integration test that upgrades the test DB to head, or the models unit home): the column exists, is nullable, is NOT indexed (assert against the live table metadata after `upgrade head`); the `Document` model exposes `file_hash` (default None); a row created with no `file_hash` round-trips NULL (the pre-phase shape).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: migration upgrade (to head) + downgrade + re-upgrade round-trip on the test DB (the existing migration-test pattern — extend it, don't fork it).
|
||||
- Coverage: **>90%** on this task's new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `0025_documents_file_hash` chains from `0024` and upgrades/downgrades cleanly on the test DB
|
||||
- [ ] `file_hash` is nullable String(64) with no index; `content_hash` (indexed, exposed) is untouched
|
||||
- [ ] `Document.file_hash` defaults to None and carries the house-style docstring distinguishing it from `content_hash`
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,51 @@
|
||||
# Task 02 — the unchanged fast path: hash first, then skip extraction, vision, and embedding
|
||||
|
||||
**Phase:** `147_sync_changed_only` · **Source:** `TODO.md:4` — "Local sources, like PDFs, are synced every time even if they haven't changed. This results in really long sync times for PDFs that are mostly pictoral. Only sync documents if they've changed since the last sync (hash the document contents to check)."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
An unchanged file (raw bytes identical to the last index) is detected at the TOP of `_index_file` — before the image/text/PDF dispatch — and takes one shared unchanged path with ZERO llm calls, ZERO PDF extraction, and ZERO vision work, while the three branches' existing unchanged semantics (summary-NULL backfill, manual-date early-return, date refresh) survive verbatim, and every newly indexed/updated document stamps its file hash.
|
||||
|
||||
## Work
|
||||
1. `app/rag/importer.py` — new helper `_file_hash(path: Path) -> str` (module level, near the other file helpers): sha256 of the raw bytes, streamed in 1 MiB chunks (`open(path, "rb")` + `hashlib.sha256()` incremental `update` in a loop — a large PDF never loads fully into memory).
|
||||
2. `app/rag/importer.py` — new helper `async def _unchanged_doc(session, doc, *, source, rel, llm, summary, raw_date, file_digest)` — the ONE shared implementation of the three near-identical per-branch unchanged blocks (text ~L928, image ~L2660, PDF ~L1303), in this exact order with UNCHANGED log wordings and counts:
|
||||
1. `summary.unchanged += 1` + `logger.info("import: unchanged source=%s path=%s", source, rel)`;
|
||||
2. **stamp** `doc.file_hash = file_digest` (a pre-phase NULL row, or a bytes-changed/content-same row, learns its hash here — see step 5);
|
||||
3. the phase-118 (A2) backfill when `doc.summary is None` — `await _store_summary(session, doc=doc, source=source, rel=rel, content=doc.content, llm=llm, summary=summary, backfill=True)` (EXACTLY what the file's bytes produced is already stored in `doc.content` for all three classes — text: the file text; image: the vision description; PDF: the final constructed content — so the backfill never re-reads the file, never re-extracts, never re-describes; the text branch's pre-phase `content` variable was the freshly-read file text — byte-identical to `doc.content` by the digest-equality invariant);
|
||||
4. the D1 early-return: `if doc.created_at_manual: return` (no write at all — the manual correction survives);
|
||||
5. the D4 date refresh: `target = normalize_doc_date(raw_date)`; `if target != doc.created_at:` set + `session.commit()` + `summary.dates_updated += 1` + the `import: date-refreshed` log line (unchanged wording) — the date may go OLDER, no monotonic guard;
|
||||
6. **final commit for the stamp**: if the row is still dirty after the date block (the stamp set `file_hash` and the date did not change) → `session.commit()` (quiet bookkeeping — no count, no log: the `unchanged` line already fired; a date-change run already committed in step 5).
|
||||
3. `app/rag/importer.py` — `_index_file` (L879): BEFORE the image/PDF/text dispatch:
|
||||
- `file_digest = _file_hash(full_path)` (every class hashes the raw bytes — images, text, PDFs alike; this is the single change signal, per the owner's "hash the document contents");
|
||||
- the row lookup moves up here: `doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))` (the branches keep their OWN lookups only if the fast path was not taken — see step 4);
|
||||
- `if raw_date is None: raw_date = file_mtime_datetime(full_path)` (the D2 fallback, read before the unchanged early-return — the branches' existing convention, now once at the top);
|
||||
- `if doc is not None and doc.file_hash == file_digest: await _unchanged_doc(session, doc, source=source, rel=rel, llm=llm, summary=summary, raw_date=raw_date, file_digest=file_digest); return` — **zero** further work: no `read_text`, no `read_bytes`, no `extract_pdf_text`, no fallback/pictorial vision, no `_replace_content_chunks`, no `_store_summary` (unless the backfill fires).
|
||||
4. `app/rag/importer.py` — the three branches, rewired minimally:
|
||||
- each branch receives `file_digest` (and the already-computed `raw_date`) from `_index_file`; their internal unchanged blocks are REPLACED by the `_unchanged_doc` call (same arguments — one code site for the semantics);
|
||||
- their create/update writes gain `doc.file_hash = file_digest` (all three — the `Document(...)` constructor for new rows, the `doc.file_hash = ...` line for updates, next to `content_hash`);
|
||||
- their existing row lookups and content-digest computations STAY (reachable only when `file_hash` is NULL — pre-phase row — or the bytes changed: the content digest then decides changed vs. the shared helper, and the helper stamps the new hash);
|
||||
- the image branch's `data = full_path.read_bytes()` and the text branch's `read_text` happen only AFTER the fast-path return point (unchanged — the dispatch is what gates them), so the fast path genuinely pays nothing beyond the streamed hash + lookup.
|
||||
5. `app/rag/importer.py` — docstrings: `_index_file`'s gains the phase-147 paragraph (the fast path: what it skips, the NULL-hash fall-through, the stamping rule); the three branches' "unchanged path" docstring lines are updated to point at the shared helper; the module docstring's A9-format paragraph is untouched.
|
||||
6. `tests/unit/` — the importer unit home (the existing `tests/unit/test_importer*.py` matrix — extend it): with a call-counting fake llm (embed + chat both counted):
|
||||
- **text unchanged:** index a file, capture the llm call count; re-run `import_sources` → `summary.unchanged == 1`, ZERO new llm calls, `summary.files == 1`, the doc's `indexed_at`/`content_hash` unchanged, `file_hash` stamped after run 1;
|
||||
- **garbage-layer PDF unchanged:** a no-text-layer PDF (the house fixture recipe) indexed with the fake vision; re-run → `summary.unchanged == 1`, ZERO new llm calls (the fallback vision did NOT re-fire — the owner's core complaint), the content row byte-identical;
|
||||
- **pictorial PDF unchanged:** a sparse-page PDF (`pdf_pictorial` on, fake vision) indexed; re-run → ZERO new llm calls (the pictorial pass did NOT re-fire);
|
||||
- **image unchanged:** re-run → ZERO new llm calls;
|
||||
- **changed file:** flip one byte → re-run → `summary.updated == 1`, the llm calls fire again, `file_hash` re-stamped, `indexed_at` bumped;
|
||||
- **pre-phase row (NULL `file_hash`):** a row with content but NULL `file_hash` (seeded directly) + an unchanged file → the branch's content-digest path runs (today's behavior — the llm may or may not be called per the digest match) AND the hash ends stamped (both the changed and the shared-helper outcomes);
|
||||
- **D1/D4 on the fast path:** an unchanged file with `created_at_manual=True` → no write at all (the date stays the manual value, no commit beyond the stamp — assert the row's date unchanged and the flag intact); an unchanged file whose mtime-derived date differs → the date refreshes (may go older), `dates_updated == 1`;
|
||||
- **summary-NULL backfill on the fast path:** an unchanged doc with `summary=None` → the backfill runs from `doc.content` (ONE `is_summary` chunk, no content re-embed — the phase-118 pin, now via the shared helper);
|
||||
- **streamed hash:** a >1 MiB temp file hashes identically to `hashlib.sha256(bytes).hexdigest()` (the chunking is transparent).
|
||||
7. The sync status/result wire (`app/api/sync.py` + `ImportSummary` fields) is UNTOUCHED — the fast path feeds the same `unchanged`/`dates_updated`/`added`/`updated` counts the UI already renders.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the matrix above — the ZERO-llm-call assertions on the second sync are the load-bearing pins (the owner's "really long sync times for PDFs that are mostly pictoral" is fixed when a pictorial PDF's second sync burns no vision).
|
||||
- Coverage: **>90%** on this task's new/modified code (`app/`); the pre-phase importer pins (progress pre-walk, `limit`, prune, the post-index passes) pass UNCHANGED.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the fast path fires on a `file_hash` match for all three classes and returns before any branch code (zero llm/extraction/vision — pinned by call counters)
|
||||
- [ ] the shared `_unchanged_doc` preserves the unchanged semantics verbatim: counts, log wordings, the backfill (from `doc.content`), the D1 early-return, the D4 refresh (may go older, commits, counts)
|
||||
- [ ] every create/update in all three branches stamps `file_hash`; the shared helper stamps NULL/changed rows too (the next sync is always fast)
|
||||
- [ ] pre-phase rows behave exactly as today on the first post-upgrade sync and are stamped
|
||||
- [ ] `content_hash`, the retriever, the progress/chunk hooks, pruning, and the per-source post-index passes are byte-identical to pre-phase
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 03 — the dedicated E2E: the idempotent second sync provably re-indexes nothing
|
||||
|
||||
**Phase:** `147_sync_changed_only` · **Source:** `TODO.md:4` — "Local sources, like PDFs, are synced every time even if they haven't changed. This results in really long sync times for PDFs that are mostly pictoral. Only sync documents if they've changed since the last sync (hash the document contents to check)."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
End-to-end proof, through the real UI + the real sync path (a real `git clone` of a local `file://` fixture repo, a real import against the mock LLM): the second sync of an unchanged source reports everything unchanged and touches no document row; a source change syncs ONLY the changed part.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_sync_changed_only.py` — the dedicated suite (one Playwright file, run in isolation; the `test_sync_button.py` module-app idiom — the module app boots with `BOR_GIT_SOURCES=file://<fixture repo>` + its own `BOR_SOURCES_DIR` on its own port (env-overridable, the house port-constant pattern), the real `git` clone prerequisite, the generous `SYNC_TIMEOUT_MS`):
|
||||
- **Fixture:** the house fixture-repo builder (the `test_sync_button.py` recipe — a local git repo with a few importable docs, e.g. the `notes/sync-fixture.md` sentinel doc + one or two more, committed) — NO PDF needed (the unit matrix covers the vision-skip proof; this E2E pins the WIRE + the UI settle line + the row-level proof, and stays fast/deterministic).
|
||||
- **Test 1 — `test_second_sync_unchanged`:** admin login (the house `login` helper); sync run 1 → wait for the settle line (the `Synced HH:MM` pattern) → capture EVERY document row's `indexed_at` + `content_hash` + `file_hash` (via `SessionLocal` + `select(Document)`, the `test_sync_button.py` DB-assertion idiom) → assert `file_hash` is non-NULL on every row (the first sync stamped them) → sync run 2 → the settle line matches the house count shape with `0 added`, `0 updated`, and the unchanged count == the run-1 document count (`fmtSyncResult` renders `N unchanged` — assert the visible `#sync-result` text) → re-query: every row's `indexed_at`/`content_hash` UNCHANGED (a re-index would bump `indexed_at` to now — the load-bearing assertion), `file_hash` unchanged.
|
||||
- **Test 2 — `test_change_syncs_only_the_change`:** from the settled state, append a NEW doc file to the fixture repo working tree + `git commit` (the house `_git` helper) → sync run 3 → the settle line shows `1 added` (and the unchanged count == the run-1 count) → DB: the new row exists with `file_hash` non-NULL; the pre-existing rows' `indexed_at`/`content_hash` unchanged.
|
||||
- **Test 3 — `test_changed_file_reindexes`:** modify the fixture sentinel doc's content (append a line) + `git commit` → sync run 4 → the settle line shows `1 updated` → DB: exactly that row's `indexed_at`/`content_hash`/`file_hash` changed; the others untouched.
|
||||
2. Every test leaves the app stopped (the module-app fixture teardown — the house pattern); the suite is hermetic (its own `BOR_SOURCES_DIR` under a tmp dir, its own port) so `--no-cov` isolation holds.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the three tests above, mock-only (the module app's LLM is the mock), real git + real import; each test asserts on the VISIBLE settle line (the owner's surface) AND the DB rows (the proof).
|
||||
- Coverage: n/a (E2E task — `app/` coverage stays >90% from tasks 01–02).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] run 2 of an unchanged source: the settle line reads `0 added · 0 updated · N unchanged` (N = the KB size) and no document row's `indexed_at`/`content_hash`/`file_hash` moved
|
||||
- [ ] a new file syncs as `1 added` without touching the other rows; a modified file syncs as `1 updated` and only its own row changes
|
||||
- [ ] every synced row carries a non-NULL `file_hash` after the first run
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` green in isolation (DB up, git on PATH — the documented prerequisite); `uv run pytest` still green (the pre-existing sync E2Es — the idempotent-second-run pins of `test_sync_button.py` etc. — pass unchanged)
|
||||
Reference in New Issue
Block a user