phase: 194_sync_pending_count
Build and Push Containers / build-and-push-app (push) Successful in 2m5s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 194 — final verification pass: all green.**

**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.

**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**

**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.

**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.

**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.

**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
2026-10-20 06:36:21 -04:00
parent ae080ac7f3
commit a45f744604
55 changed files with 7610 additions and 179 deletions
+163 -21
View File
@@ -103,7 +103,16 @@ carries the phase-64 per-file progress — ``current_file`` (the
``source/relative/path`` the import is processing right now) plus
``files_done`` / ``files_total`` — null/0/0 before the import starts
(clone/pull reports no file yet) and in terminal states, which clear
``current_file`` but keep the run's final counts.
``current_file`` but keep the run's final counts. What the pair COUNTS
changed in phase 194: on a progress caller (the UI sync is the only
one) it is the position in the pre-scan's PREDICTED change set —
``files_total`` the number of files that will actually be re-indexed,
``files_done`` the 1-based position in that set — which is what makes
the Sources page's ``N remaining`` say "10 remaining" on a ~10,000-file
KB with 10 changes (the owner report of 2026-10-19) instead of counting
the walk. ``BOR_SYNC_PENDING_COUNT=0`` restores the phase-193 walk
position byte-for-byte; the terminal ``detail`` counts stay the exact
surface either way.
Phase 134 (task 01) makes the machine FIVE-state: ``cancelled`` joins
``success`` / ``failed`` as a TERMINAL state — it carries
@@ -131,7 +140,14 @@ checks the event at the pipeline boundaries via
set): the top of the per-source loop (before each clone/pull or
local-dir check), immediately before ``import_sources``, and
immediately before the overview + summaries span (the
``added + updated > 0`` block). Consequence a future reader must
``added + updated > 0`` block). Phase 194 (task 02) adds one more
await INSIDE the import span for the same shape: the pre-scan runs in
a ``to_thread`` worker, so a cancel parked there unblocks the RUN at
once (the ``cancelled`` terminal, nothing indexed) while the scan
thread finishes the session of its own — the phase-134/179 split, the
worker not being interruptible, pinned by
``tests/integration/test_sync_pending_count.py`` (importer side) and
this module's cancel-during-scan pin. Consequence a future reader must
know: a git subprocess already running at the cancel moment runs to
completion (Python threads are not interruptible — since phase 179
it occupies a ``to_thread`` worker rather than the loop, so it no
@@ -188,8 +204,17 @@ background task must die in state, never as an unobserved exception
Phase 98 (task 01) adds the run's PHASE machine — ``phase`` is the
pipeline stage the run is in: the model-check + clone/pull prelude
reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
(set immediately before ``import_sources``), ``"overview"`` (set
reports ``null`` (the bare "Syncing…" label stays), then ``"scan"``
(phase 194, task 02 — set immediately before ``import_sources``,
REPLACING the pre-phase ``"import"`` set: what happens first in that
call is the pre-scan that predicts the change set, and the label the
task-03 branch renders for this phase is ``Scanning… (n/total)``), then
``"import"`` (flipped by
the FIRST per-file ``progress`` call while the phase is still
``"scan"`` — the flip lives in the runner's own ``_hook`` closure, so
the importer gains no extra callback; with
``BOR_SYNC_PENDING_COUNT=0`` no scan report fires at all and the flip
is simply the first file), ``"overview"`` (set
before ``regenerate_overview`` — a changed KB only), and
``"summaries"`` (set before a folder-summary generation — the
changed-KB incremental regeneration (phase 127, task 03 — the changed
@@ -259,7 +284,12 @@ sampled through the ONE :func:`_sample_rss` — at the run start, at every
``phase`` transition, in every terminal, and at most once per
``_RSS_SAMPLE_INTERVAL_S`` from the per-file progress hook, so a 27k-file
walk costs ~600 ``/proc`` reads and not 27k — LOCKED A4), and
``GET /api/sync/status`` reports both keys ADDITIVELY. The terminals keep
``GET /api/sync/status`` reports both keys ADDITIVELY. Phase 194
(task 02) extends both halves of that rule to the new stage: the
``"scan"`` set before ``import_sources`` and the ``scan``→``import``
flip are phase TRANSITIONS and FORCE the read, while the pre-scan's own
per-file hook samples UNFORCED (throttled) — a 10,000-file scan costs
one ``/proc`` read per interval, never one per file. The terminals keep
the FINAL reading next to the final counts (the phase-64 convention), so
the page that started the run answers the incident's question without a
shell. A reading the platform cannot know stays ``null`` on the wire (the
@@ -323,11 +353,33 @@ class SyncStatus:
outside the import phase — clone/pull first, terminal states
after); ``files_done`` / ``files_total`` carry the hook's
done/total position and survive a terminal state (the run's last
position is useful context next to the error).
position is useful context next to the error). Since phase 194 the
pair is the hook's PENDING position (phase 194: the pre-scan's
predicted change set — ``files_total`` the files that will actually
be re-indexed), not the walk position; ``BOR_SYNC_PENDING_COUNT=0``
restores the walk position and every other caller (the CLI, the
upload job) is unaffected.
Phase 194 (task 02) scan fields: ``scan_done`` / ``scan_total``
carry the pre-scan's position — the 1-based file whose hash+read
just completed over the combined importable count the scan walks
(the importer's ``on_scan_progress`` hook). They are 0/0 before the
scan starts and on the ``BOR_SYNC_PENDING_COUNT=0`` rollback (no
scan, so no report), reset with the run like every other field, and
KEPT at every terminal beside the final file counts (the phase-64
keep-final-values convention — the scan the owner watched is still
on the page after the run). ``current_file`` stays ``null`` through
the phase: the scan names no file, it only counts. The degenerate
empty knowledge base is the honest edge — the scan predicts nothing
so no report fires and no ``progress`` call ever arrives: the phase
stays ``"scan"`` until the next transition or the terminal (which
clears it), and the label is the bare phase name with no count part
(the UI's ``total === 0`` guard).
Phase 98 (task 01) phase fields: ``phase`` is the pipeline stage
(null in the model-check + clone/pull prelude and in terminal
states — the module docstring's phase machine); while in the
states — the module docstring's phase machine, whose stages are now
``scan`` → ``import`` → ``overview`` → ``summaries``); while in the
``"summaries"`` phase, ``current_summary`` is the source /
``source/folder_path`` the generator is summarizing right now and
``summaries_done`` / ``summaries_total`` carry the generator's
@@ -393,10 +445,20 @@ class SyncStatus:
current_file: str | None = None
files_done: int = 0
files_total: int = 0
# Phase 194 (task 02): the PRE-SCAN's position — the 1-based file
# whose hash+read just completed and the combined importable count
# the scan walks (the importer's on_scan_progress hook). 0/0 before
# the scan starts (and on the BOR_SYNC_PENDING_COUNT=0 rollback);
# reset with the run and kept at every terminal beside the final
# file counts (the phase-64 keep-final-values convention).
scan_done: int = 0
scan_total: int = 0
# Phase 98 (task 01): the phase machine — the pipeline stage the
# run is in and, while in the folder-summary phase, the folder
# being summarized plus the hook's done/total position.
phase: Literal["import", "overview", "summaries"] | None = None
# being summarized plus the hook's done/total position. Phase 194
# (task 02) adds the "scan" stage (the pre-scan that predicts the
# run's change set — the module docstring's phase machine).
phase: Literal["scan", "import", "overview", "summaries"] | None = None
current_summary: str | None = None
summaries_done: int = 0
summaries_total: int = 0
@@ -433,13 +495,15 @@ _task: asyncio.Task[None] | None = None
# starts, so a stale event can never bleed into the next run.
_cancel_event = asyncio.Event()
#: Phase 187 (task 03, D3/A4): how often the PER-FILE progress hook may
#: re-read the process memory, in seconds. That hook fires once per
#: file, so without this guard a 27k-file walk would cost 27k
#: ``/proc/self/status`` reads; with it the same walk costs ~600 and the
#: operator watching the page still sees the number move (PLAN §7.4's
#: never-stale contract). Every other sampling point (the run start, the
#: ``phase`` transitions, the terminals) FORCES the read instead.
#: Phase 187 (task 03, D3/A4): how often the PER-FILE progress hooks may
#: re-read the process memory, in seconds. Those hooks fire once per
#: file — the import's ``progress`` AND, since phase 194 (task 02), the
#: pre-scan's ``on_scan_progress`` — so without this guard a 27k-file
#: walk would cost 27k ``/proc/self/status`` reads; with it the same walk
#: costs ~600 and the operator watching the page still sees the number
#: move (PLAN §7.4's never-stale contract). Every other sampling point
#: (the run start, the ``phase`` transitions including the ``scan`` set
#: and its flip to ``import``, the terminals) FORCES the read instead.
_RSS_SAMPLE_INTERVAL_S = 5.0
#: ``time.monotonic()`` of the last RSS read; ``-inf`` means "never", so
@@ -503,7 +567,21 @@ def sync_status() -> dict[str, Any]:
import is processing right now — null during the clone/pull phase
and in terminal states; ``files_done`` / ``files_total`` carry the
hook's position (0/0 idle) and keep their final values in the
terminal. ``current_chunk`` / ``chunks_total`` (phase 129) are the
terminal — the position in the pre-scan's PREDICTED change set on a
progress caller (phase 194), the walk position under
``BOR_SYNC_PENDING_COUNT=0``. ``scan_done`` / ``scan_total`` (phase
194) are the PRE-SCAN's position: the 1-based file whose
hash+compare just ran over the combined importable count the scan
walks. They are 0/0 before the scan starts — the prelude, a run on
the ``0`` rollback, and every run that never happened — and in a
terminal they keep the FINAL scan values beside the final file
counts (the phase-64 keep-final-values convention); ``current_file``
stays null while ``phase`` is ``"scan"``, because the scan names no
file. Both keys are ADDITIVE — placed after ``files_total``, every
pre-existing key keeping its name, type and position. The degenerate
empty knowledge base reports the phase without ever reporting a
position (0/0 — the UI's ``total === 0`` guard renders no count
part). ``current_chunk`` / ``chunks_total`` (phase 129) are the
in-flight file's embed-chunk position (batch granularity — the
first chunk of the in-flight batch over the file's total content
chunks): 0/0 before the first batch, in the prelude/overview/
@@ -551,6 +629,14 @@ def sync_status() -> dict[str, Any]:
"current_file": _status.current_file,
"files_done": _status.files_done,
"files_total": _status.files_total,
# Phase 194 (task 02): the pre-scan's position (the importer's
# on_scan_progress hook) — 0/0 before the scan starts and on the
# BOR_SYNC_PENDING_COUNT=0 rollback, the FINAL pair kept in every
# terminal beside the file counts above (the keep-final-values
# convention). Additive: every pre-existing key keeps its name,
# type and position.
"scan_done": _status.scan_done,
"scan_total": _status.scan_total,
# Phase 129 (task 03): the in-flight file's embed-chunk
# position — the dataclass defaults make idle 0/0, and the
# runner clears it at every point the import phase ends
@@ -672,6 +758,14 @@ async def _run_sync() -> None:
so a long clone no longer stalls the rest of the HTTP surface (the
module docstring's freeze paragraph); the run's own sequencing is
byte-for-byte the pipeline above.
Phase 194 (task 02): the phase set before ``import_sources`` is
``"scan"`` (not ``"import"``), because that call begins with the
off-loop pre-scan that predicts the change set; the FIRST per-file
``progress`` call flips it to ``"import"`` inside ``_hook``. Both
transitions force the memory sample, the scan's own per-file hook
does not (LOCKED A4), and ``scan_done`` / ``scan_total`` reset with
the run and survive every terminal beside the file counts.
"""
_status.state = "running"
_status.started_at = datetime.now(UTC)
@@ -683,6 +777,11 @@ async def _run_sync() -> None:
_status.current_file = None
_status.files_done = 0
_status.files_total = 0
# Phase 194 (task 02): the pre-scan's position resets with the run —
# 0/0 until the scan reports its first file (and 0/0 for the whole
# run on the BOR_SYNC_PENDING_COUNT=0 rollback, which scans nothing).
_status.scan_done = 0
_status.scan_total = 0
# Phase 98 (task 01): the phase fields reset with the run — no
# phase until the import starts (the model-check + clone/pull
# prelude reports null), no folder until the summary span starts.
@@ -834,6 +933,17 @@ async def _run_sync() -> None:
# run-start reset alone cannot cover it).
_status.images_done = 0
_status.images_total = 0
# Phase 194 (task 02): the scan→import FLIP. The phase is
# set to "scan" immediately before import_sources, and the
# first file the walk actually reaches is where the running
# label stops being a scan — no extra callback reaches the
# importer, the flip lives in this existing per-file hook.
if _status.phase == "scan":
_status.phase = "import"
# A phase TRANSITION is a forced sampling point (phase
# 187), so this one read is not the throttled per-file
# one the tail of this closure does.
_sample_rss(force=True)
# Phase 187 (task 03): the run's memory, sampled THROTTLED
# (unforced) — this hook fires once per file, so a 27k-file
# walk reads /proc ~600 times instead of 27k (LOCKED A4)
@@ -865,10 +975,30 @@ async def _run_sync() -> None:
_status.images_done = done
_status.images_total = total
# Phase 98 (task 01): the phase machine — the import phase
# starts NOW (the model-check + clone/pull prelude above
# reported ``phase: null``).
_status.phase = "import"
# Phase 194 (task 02): the PRE-SCAN's position — the importer's
# ``on_scan_progress`` hook, fired once per importable file while
# the run's change set is still being PREDICTED (before the walk),
# with (position_1based, scan_total). It runs on the scan's
# WORKER thread, so this closure only assigns fields on the
# module's own status object — the house hook contract that makes
# each call a single GIL-atomic store (phase 179). The memory
# sample is UNFORCED here: this fires once per file, exactly like
# ``_hook`` above, so it costs one /proc read per interval and
# never one per file (LOCKED A4).
def _scan_hook(scan: int, total: int) -> None:
_status.scan_done = scan
_status.scan_total = total
_sample_rss()
# Phase 98 (task 01) + phase 194 (task 02): the phase machine —
# the SCAN phase starts NOW (the model-check + clone/pull prelude
# above reported ``phase: null``). What runs first inside
# ``import_sources`` is the pre-scan that predicts the change set,
# and the label the owner reads through it is "Scanning…"
# (phase 193's "import" set moved to the first ``_hook`` call
# above — the flip), so the count the page shows is never a scan
# wearing the work's shape.
_status.phase = "scan"
# Phase 187 (task 03): the phase transitions are the forced
# sampling points — each stage of the run begins with its own
# reading on the status (the per-file hook throttles).
@@ -885,6 +1015,7 @@ async def _run_sync() -> None:
progress=_hook,
on_chunk_progress=_chunk_hook,
on_image_progress=_image_hook, # phase 152 (task 03)
on_scan_progress=_scan_hook, # phase 194 (task 02)
ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
doc_dates_by_root=doc_dates_by_root,
@@ -1034,6 +1165,10 @@ async def _run_sync() -> None:
_status.current_file = None # phase 64: keep the final counts
_status.phase = None # phase 98: keep the final summary counts
_status.current_summary = None
# Phase 194 (task 02): the scan position is NOT cleared here —
# like the file counts it keeps its final values beside them
# (the keep-final-values convention), so the page reloaded after
# the run still says how big the predicted change set was.
# Phase 129 (task 03): the chunk position is a sub-position of
# a file that's done — it clears in the terminal (unlike the
# FILE counts, which keep their final values for the result
@@ -1091,6 +1226,10 @@ async def _run_sync() -> None:
_status.current_file = None # keep the final counts
_status.phase = None # keep the final summary counts
_status.current_summary = None
# Phase 194 (task 02): the scan position survives the cancel
# too — where a cancel landed DURING the pre-scan it is the
# run's only number, and the log line above keeps the file
# pair beside it (the keep-final-values convention).
# Phase 129 (task 03): the chunk position is a sub-position
# of a file that's done — it clears in EVERY terminal, like
# success and failed (the convention split, pinned).
@@ -1138,6 +1277,9 @@ async def _run_sync() -> None:
_status.current_file = None # phase 64: keep the final counts
_status.phase = None # phase 98: keep the final summary counts
_status.current_summary = None
# Phase 194 (task 02): the scan position is kept here as well —
# a run that died mid-scan keeps the position it reached beside
# the final file counts (the convention, every terminal).
# Phase 129 (task 03): the chunk position clears in the
# failed terminal too (the convention split — the FILE counts
# keep their final values next to the error, the sub-position
+30
View File
@@ -524,6 +524,22 @@ class Settings(BaseSettings):
#: mid-prune leaves a PREFIX pruned, and the next sync converges
#: because ``seen`` is recomputed from the filesystem every run.
sync_prune_batch: int = 500
#: Predict the run's change set BEFORE the walk, so the sync's running
#: count is the count of the WORK and not of the walk (phase 194,
#: ``BOR_SYNC_PENDING_COUNT``). ``1`` (default): ``import_sources``
#: pre-scans the importable files (one streamed sha256 + one scalar
#: ``documents.file_hash`` read each, off the event loop) and the
#: per-file ``progress`` hook then reports ``done``/``total`` as the
#: position in that PENDING set — the owner's "1216 remaining" fix.
#: ``0`` = the exact phase-193 behaviour (the stats-only pre-walk and
#: ``done = summary.files``, i.e. "every file left to WALK") — the
#: operator's no-redeploy rollback if the extra read cost ever bites.
#: A negative value is a typo — the validator fails startup loudly
#: (the ``agent_max_rounds`` pattern). Read per run (the importer reads
#: ``llm.settings`` once per sync, so a saved override applies to the
#: NEXT sync) and only by a caller that passes ``progress`` — the CLI
#: and the upload job pay nothing.
sync_pending_count: int = 1
# --- Image documents (phase 122: standalone images as documents) ---
#: Master switch for image-document indexing (phase 122,
@@ -930,6 +946,20 @@ class Settings(BaseSettings):
)
return v
@field_validator("sync_pending_count")
@classmethod
def _sync_pending_count_non_negative(cls, v: int) -> int:
"""``0`` is the phase-193 rollback (count the walk, not the work) —
a negative value is a typo that no branch of the importer can mean,
so it fails startup naming the field (the ``agent_max_rounds``
pattern, phase 194)."""
if v < 0:
raise ValueError(
"sync_pending_count must be >= 0 (0 = count the whole walk, "
"the pre-phase-194 behaviour)"
)
return v
@field_validator("chat_image_max_mb")
@classmethod
def _chat_image_max_mb_positive(cls, v: int) -> int:
+3
View File
@@ -304,6 +304,9 @@ CATALOG: dict[str, SettingMeta] = {
"sync_prune_batch": SettingMeta(
"sync_prune_batch", "Import & sources", "Sync prune batch"
),
"sync_pending_count": SettingMeta(
"sync_pending_count", "Import & sources", "Sync pending count"
),
# --- Images ---
"images": SettingMeta("images", "Images", "Images"),
"image_extensions": SettingMeta(
+287 -30
View File
@@ -332,14 +332,27 @@ convention, phase 133).
(phase 64, task 01) reporting the file being processed right now, an
optional ``on_chunk_progress`` callback (phase 129, task 02)
reporting the in-flight file's embedding-chunk position at batch
granularity, and an optional ``on_image_progress`` callback (phase
152, task 02) reporting the in-flight PDF's vision-image position —
granularity, an optional ``on_image_progress`` callback (phase 152,
task 02) reporting the in-flight PDF's vision-image position —
immediately before each of the PDF's vision describe calls, with the
image's 1-based position in the file's ONE continuous vision counter
and the span's count (see the function docstring).
and the span's count — and an optional ``on_scan_progress`` callback
(phase 194, task 01) reporting the PRE-SCAN's position while the run's
change set is predicted. The ``progress`` pair is the pending position
(``total`` = the size of the predicted change set, ``done`` = the 1-based
position in it) rather than the walk position, which is what makes the
Sources page's ``N remaining`` count the WORK and not the WALK: a
~10,000-file sync that changes 10 files says "10 remaining", not
"1216 remaining" (the owner report of 2026-10-19). ``0`` in
``BOR_SYNC_PENDING_COUNT`` restores the phase-193 walk position exactly.
All four callbacks follow the same "only when provided" rule — a caller
that passes nothing pays the pass, the read and the cost of none of them,
and a caller that passes no ``progress`` runs no pre-scan at all (the
CLI, the upload job: byte-identical). See the function docstring.
"""
from __future__ import annotations
import asyncio
import hashlib
import logging
import os
@@ -753,6 +766,125 @@ def _doc_row_stmt(source: str, rel: str) -> Select[tuple[Document]]:
)
def _pending_hash_stmt(source: str, rel: str) -> Select[tuple[str | None]]:
"""The pre-scan's ONE-column metadata read (phase 194, task 01).
``select(Document.file_hash)`` is a SCALAR projection, not an entity
load: no ``Document`` is ever constructed, nothing lands in the run's
``identity_map``, and the pre-scan therefore retains nothing — which is
why it needs no ``defer``/``load_only`` (there is no row to defer) and
why it does NOT touch :func:`_doc_row_stmt`, which stays the importer's
ONLY row load (phase 185). The predicate is the unique
``(source, path)`` constraint (``app/models.py``, the same lookup the
fast path uses), so the read costs one index probe per file — the
``file_hash`` column itself stays unindexed by design.
"""
return select(Document.file_hash).where(
Document.source == source, Document.path == rel
)
def _prescan_pending(
sources: Sequence[Path],
extensions: frozenset[str],
image_extensions: frozenset[str],
ignore_by_root: dict[str, list[str]] | None,
include_hidden_by_root: dict[str, bool] | None,
on_scan_progress: Callable[[int, int], None] | None,
session: Session | None = None,
) -> tuple[int, dict[tuple[str, str], int]]:
"""Predict the run's change set BEFORE the walk (phase 194, D1).
Returns ``(scan_total, pending)``: the combined count of importable
files across *sources* — the SAME number the phase-64 stats-only pre-walk
computed, under the SAME ``iter_importable_files`` rules (the caller
passes the exact *extensions* / *image_extensions* / per-root ``ignore``
and ``include_hidden`` values the processing loop uses, so a file the
loop cannot see is never predicted) — and ``pending``, the map of
``(source, rel)`` to that file's 1-based position in the predicted change
set, walk order.
Per file: :func:`_file_hash` (the streamed 1 MiB-chunk sha256 of phase
147 — the raw bytes, the sync's single change signal) plus ONE
:func:`_pending_hash_stmt` scalar read, and the file is pending when the
stored hash is ``NULL`` (no row yet, or a pre-phase-147 row) or differs
from the digest — the SAME predicate the fast path applies at processing
time, so a pre-phase NULL row is predicted changed AND really re-indexed
and stamped by the loop: the prediction never lies about the stored-hash
contract. A file the digest cannot reach (deleted between the walk and
the hash) is predicted changed too: the loop's own walk no longer sees
it, so the count can hold one short, which is honest — the terminal
counts line is the exact surface either way.
``session`` follows the importer's own convention (tests pass one, a
private one is opened and closed otherwise). Deliberately NOT the RUN's
session: this function executes inside an :func:`asyncio.to_thread`
worker (see :func:`import_sources`), and a ``Session`` is not
thread-safe — a cancel at the span's await lets this worker run to
completion while the run has already moved on to its terminal and closed
its own session. Reading through a session this span owns is what makes
that edge safe; the projection loads no entity, so even this session
keeps nothing.
*on_scan_progress* is called — only when provided — once per file with
``(position_1based, scan_total)``, AFTER that file's hash+read (so the
position counts files actually scanned and reaches the total exactly when
the scan does). It runs on the WORKER thread: the house hook contract is
that these hooks only assign fields on the caller's own status object,
which makes each call a single GIL-atomic store, and there is no
try/except around it for the same reason as the three sibling hooks.
"""
owns_session = session is None
if session is None:
session = SessionLocal()
try:
# (a) the per-root file lists + the combined length, so the scan's
# denominator is known before the first callback. Directory stats
# only — no file read, no database statement (the phase-64 shape).
per_root: list[tuple[Path, list[Path]]] = [
(
root,
iter_importable_files(
root,
extensions,
ignore=_ignore_for_root(root, ignore_by_root),
include_hidden=_include_hidden_for_root(
root, include_hidden_by_root
),
image_extensions=image_extensions,
),
)
for root in sources
]
scan_total = sum(len(files) for _, files in per_root)
pending: dict[tuple[str, str], int] = {}
position = 0
for root, files in per_root:
source = root.name
for path in files:
rel = path.relative_to(root).as_posix()
try:
digest = _file_hash(path)
except OSError as e: # vanished mid-scan — predict the change
logger.warning(
"import: pre-scan unreadable source=%s path=%s — %s",
source,
rel,
e,
)
digest = ""
stored = session.scalar(_pending_hash_stmt(source, rel))
if stored is None or stored != digest:
pending[(source, rel)] = len(pending) + 1
position += 1
if on_scan_progress is not None:
on_scan_progress(position, scan_total)
return scan_total, pending
finally:
if owns_session:
session.close()
async def import_sources(
sources: list[Path],
llm: Embedder,
@@ -763,6 +895,7 @@ async def import_sources(
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None,
on_image_progress: Callable[[int, int], None] | None = None,
on_scan_progress: Callable[[int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
@@ -776,15 +909,53 @@ async def import_sources(
``progress`` (phase 64, task 01) is an optional per-file hook called
once per importable file, immediately before that file's
``_index_file`` — with ``(source, rel_posix_path, done, total)``: the
same POSIX *rel* the document rows use, ``done`` = the 1-based index of
the current file **across all sources**, and ``total`` = the combined
pre-walk count of importable files across all *sources* roots. The
pre-walk (same extension/exclusion rules, directory stats only, no file
reads) happens **only when *progress* is provided**: callers passing
nothing pay no extra walk and behave exactly as before. Under
``limit``, the hook still fires per processed file only — ``done`` never
exceeds the limit, but ``total`` stays the full pre-walk count (an
incomplete walk must not misreport the denominator).
same POSIX *rel* the document rows use, plus the position pair the UI's
running count consumes. The pair's MEANING is the phase-194 switch
``llm.settings.sync_pending_count`` (read ONCE before the source loop —
see the pre-scan paragraph below):
* switch truthy (the default): ``total`` = the size of the pre-scan's
PENDING set (the files predicted to be re-indexed) and ``done`` = the
1-based position in THAT set. The position advances for every file in
the pending set — whether or not the file turns out changed at
processing time — so count and denominator stay on the SAME set and
the arithmetic never runs away; a file NOT in the set does not advance
it, which is what keeps the label holding at ``K remaining`` while the
walk moves over the unchanged ``N − K``.
* switch ``0``: the exact phase-193 walk position — ``done`` = the
1-based index of the current file **across all sources** and ``total``
= the combined pre-walk count of importable files (the full rollback).
In both states the count-only pass happens **only when *progress* is
provided**: callers passing nothing pay no extra pass and behave exactly
as before (the CLI, the upload job). Under ``limit``, the hook still
fires per processed file only — ``done`` never exceeds the limit, but
``total`` stays the full count (an incomplete walk must not misreport
the denominator).
The pair is a PREDICTION, and its drift edges are bounded (phase 194
A1): a file the scan predicted and that reverted to unchanged before the
loop reached it still advances the count (it was counted in ``total``,
so the count still reaches 0); a file changed AFTER the scan is indexed
for real but advances nothing (``total`` never counted it), and the
UI's ``total - done > 0`` guard hides that excess — never a negative and
never a stale ``0 remaining``. The terminal ``counts_line`` ("N added ·
M updated …") stays the exact surface, untouched.
``on_scan_progress`` (phase 194, task 01) is an optional SCAN-position
hook for the pre-scan itself: while the pending set is being predicted it
is called — once per importable file, after that file's hash+read — with
``(position_1based, scan_total)``, the file's 1-based position in the
scan and the combined importable count across all *sources* roots (the
number the phase-64 stats pre-walk used to compute). It carries counts
only, never the file identity (the sibling convention), and it runs on
the span's WORKER thread — the house rule that these hooks only assign
fields is what makes each call a single GIL-atomic store. Same "only
when provided" rule as ``progress``, ``on_chunk_progress`` and
``on_image_progress``, with one addition: the pre-scan it reports on is
gated on *progress* too, so a caller passing ``on_scan_progress`` alone
(``progress=None``) runs no scan, fires no callback and stays
byte-identical — the scan exists only to serve the running counts.
``on_chunk_progress`` (phase 129, task 02) is an optional
chunk-position hook for the file currently being indexed: while that
@@ -923,6 +1094,45 @@ async def import_sources(
(``0`` = never — the exact pre-phase behaviour, the operator's
no-redeploy rollback). The cadence changes no count, no log line and no
committed byte: see the safety argument at the reset site.
The pre-scan (phase 194, task 01): with the hook on and
``llm.settings.sync_pending_count`` truthy, the count-only pass is no
longer a stats walk but :func:`_prescan_pending` — every importable file
hashed once and compared once against its stored ``file_hash``, giving
the PENDING set the ``progress`` pair counts. Three costs the phase
accepted, deliberately:
* **off the loop** (the phase-179 rule): the scan runs in ONE
:func:`asyncio.to_thread` span. Ten thousand streamed hashes plus ten
thousand one-column SELECTs inline would park the single uvicorn loop
for seconds — the exact defect class
``tests/integration/test_sync_git_freeze.py`` pins for the git clone,
and the ``GET /api/sync/status`` poll the UI runs every 2 s is the
victim. A cancel registered at the span's await interrupts the RUN
(phase 134: it lands the ``cancelled`` terminal), while the worker
itself runs to completion — threads are not interruptible, which is
also why the span reads through a session of its own instead of the
run's (see :func:`_prescan_pending`).
* **one extra read + hash + scalar SELECT per importable file per run**
(A3): the second read of the same file is served from the page cache,
and the scalar read is a unique ``(source, path)`` index probe.
``BOR_SYNC_PENDING_COUNT=0`` buys the whole cost back and restores the
phase-193 walk-position semantics byte-for-byte; a negative value
refuses startup naming the field.
* **the file is hashed twice**, here and again in ``_index_file``. That
is deliberate: the pre-scan's digest is the PREDICTION, the loop's is
the TRUTH — a file edited between the two is still detected and
re-indexed. Dropping the second hash would make the count a claim
about bytes nobody verified.
Everything else in the run is untouched by the scan: ``summary.files``
(the ``import: summary … files=`` count of every file the loop
processed), ``seen``, the prune post-pass, the date refresh and every log
line keep their exact shape — the pre-scan builds ``pending``, never
``seen``, and the loop keeps building ``seen`` from disk exactly as
before. The degenerate empty knowledge base is the phase-64 shape: scan
total ``0``, no callback fires, the pending set is empty and ``total``
is ``0`` (the UI's ``total === 0`` guard then renders no count part).
"""
if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1")
@@ -937,30 +1147,68 @@ async def import_sources(
# reads ``llm.settings`` per RUN, never per file). ``0`` disables the
# release entirely.
reset_every = llm.settings.sync_session_reset_files
# Phase 194 (task 01): the pending-count switch, read ONCE here — before
# the source loop — so a settings overlay saved mid-run cannot change the
# MEANING of the hook's ``done``/``total`` pair between two files of the
# same walk (the phase-184 read-once discipline). ``0`` = the exact
# phase-193 walk-position semantics, the operator's no-redeploy rollback.
count_pending = bool(llm.settings.sync_pending_count)
owns_session = session is None
if session is None:
session = SessionLocal()
seen: set[tuple[str, str]] = set()
source_names: set[str] = set()
# phase 64 (task 01): the hook's combined denominator, walked with the
# exact same rules as the processing loop below (directory stats only,
# no file reads). Skipped entirely for ``progress=None`` callers — no
# extra pass, byte-identical behaviour and cost.
# phase 64 (task 01): the hook's denominator. phase 194 (task 01): what
# the number MEANS follows ``count_pending`` — the size of the predicted
# change set, or the count of every importable file. Either way the pass
# that computes it is skipped entirely for ``progress=None`` callers (no
# extra cost, byte-identical behaviour) and it is INSIDE the try so the
# session is closed on every exit path.
total = 0
if progress is not None:
for root in sources:
total += len(
iter_importable_files(
root,
llm.settings.import_extension_set,
ignore=_ignore_for_root(root, ignore_by_root),
include_hidden=_include_hidden_for_root(
root, include_hidden_by_root
),
image_extensions=image_exts,
)
)
# (source, rel) -> 1-based position in the PENDING set; ``None`` = the
# phase-193 walk-position semantics (the switch ``0`` branch below).
pending: dict[tuple[str, str], int] | None = None
# the position the hook last reported: a file the scan did not predict
# holds it, which keeps ``total - done`` honest while the walk moves over
# the unchanged files.
pending_done = 0
try:
if progress is not None:
if count_pending:
# THE PRE-SCAN (phase 194, D1) — it REPLACES the stats-only
# pre-walk, so there is no third walk. ONE off-loop span
# (the phase-179 rule: a 10,000-file hash+SELECT pass inline
# would park the single uvicorn loop and freeze the status
# poll the UI runs every 2 s). The scan total is the scan
# hook's own business; the run's denominator is the pending
# set the scan returns.
_scan_total, pending = await asyncio.to_thread(
_prescan_pending,
sources,
llm.settings.import_extension_set,
image_exts,
ignore_by_root,
include_hidden_by_root,
on_scan_progress,
)
total = len(pending)
else:
# ``BOR_SYNC_PENDING_COUNT=0`` — the phase-193 rollback,
# byte-for-byte: the hook's combined denominator, walked with
# the exact same rules as the processing loop below (directory
# stats only, no file reads).
for root in sources:
total += len(
iter_importable_files(
root,
llm.settings.import_extension_set,
ignore=_ignore_for_root(root, ignore_by_root),
include_hidden=_include_hidden_for_root(
root, include_hidden_by_root
),
image_extensions=image_exts,
)
)
for root in sources:
if not root.is_dir():
logger.warning("import: source dir not found, skipping: %s", root)
@@ -1036,7 +1284,16 @@ async def import_sources(
# file that then errors or turns out unchanged was
# already the "current file". No try/except around the
# call: the hooks in this repo only assign fields.
progress(source, rel, summary.files, total)
# phase 194 (task 01): the pair's meaning follows the
# switch read before the loop — the position in the
# pending set, or the pre-phase-194 walk position.
if pending is not None:
predicted = pending.get((source, rel))
if predicted is not None:
pending_done = predicted
progress(source, rel, pending_done, total)
else:
progress(source, rel, summary.files, total)
try:
await _index_file(
session, source=source, rel=rel, full_path=path, llm=llm,