phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
+163
-21
@@ -103,7 +103,16 @@ carries the phase-64 per-file progress — ``current_file`` (the
|
||||
``source/relative/path`` the import is processing right now) plus
|
||||
``files_done`` / ``files_total`` — null/0/0 before the import starts
|
||||
(clone/pull reports no file yet) and in terminal states, which clear
|
||||
``current_file`` but keep the run's final counts.
|
||||
``current_file`` but keep the run's final counts. What the pair COUNTS
|
||||
changed in phase 194: on a progress caller (the UI sync is the only
|
||||
one) it is the position in the pre-scan's PREDICTED change set —
|
||||
``files_total`` the number of files that will actually be re-indexed,
|
||||
``files_done`` the 1-based position in that set — which is what makes
|
||||
the Sources page's ``N remaining`` say "10 remaining" on a ~10,000-file
|
||||
KB with 10 changes (the owner report of 2026-10-19) instead of counting
|
||||
the walk. ``BOR_SYNC_PENDING_COUNT=0`` restores the phase-193 walk
|
||||
position byte-for-byte; the terminal ``detail`` counts stay the exact
|
||||
surface either way.
|
||||
|
||||
Phase 134 (task 01) makes the machine FIVE-state: ``cancelled`` joins
|
||||
``success`` / ``failed`` as a TERMINAL state — it carries
|
||||
@@ -131,7 +140,14 @@ checks the event at the pipeline boundaries via
|
||||
set): the top of the per-source loop (before each clone/pull or
|
||||
local-dir check), immediately before ``import_sources``, and
|
||||
immediately before the overview + summaries span (the
|
||||
``added + updated > 0`` block). Consequence a future reader must
|
||||
``added + updated > 0`` block). Phase 194 (task 02) adds one more
|
||||
await INSIDE the import span for the same shape: the pre-scan runs in
|
||||
a ``to_thread`` worker, so a cancel parked there unblocks the RUN at
|
||||
once (the ``cancelled`` terminal, nothing indexed) while the scan
|
||||
thread finishes the session of its own — the phase-134/179 split, the
|
||||
worker not being interruptible, pinned by
|
||||
``tests/integration/test_sync_pending_count.py`` (importer side) and
|
||||
this module's cancel-during-scan pin. Consequence a future reader must
|
||||
know: a git subprocess already running at the cancel moment runs to
|
||||
completion (Python threads are not interruptible — since phase 179
|
||||
it occupies a ``to_thread`` worker rather than the loop, so it no
|
||||
@@ -188,8 +204,17 @@ background task must die in state, never as an unobserved exception
|
||||
|
||||
Phase 98 (task 01) adds the run's PHASE machine — ``phase`` is the
|
||||
pipeline stage the run is in: the model-check + clone/pull prelude
|
||||
reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
|
||||
(set immediately before ``import_sources``), ``"overview"`` (set
|
||||
reports ``null`` (the bare "Syncing…" label stays), then ``"scan"``
|
||||
(phase 194, task 02 — set immediately before ``import_sources``,
|
||||
REPLACING the pre-phase ``"import"`` set: what happens first in that
|
||||
call is the pre-scan that predicts the change set, and the label the
|
||||
task-03 branch renders for this phase is ``Scanning… (n/total)``), then
|
||||
``"import"`` (flipped by
|
||||
the FIRST per-file ``progress`` call while the phase is still
|
||||
``"scan"`` — the flip lives in the runner's own ``_hook`` closure, so
|
||||
the importer gains no extra callback; with
|
||||
``BOR_SYNC_PENDING_COUNT=0`` no scan report fires at all and the flip
|
||||
is simply the first file), ``"overview"`` (set
|
||||
before ``regenerate_overview`` — a changed KB only), and
|
||||
``"summaries"`` (set before a folder-summary generation — the
|
||||
changed-KB incremental regeneration (phase 127, task 03 — the changed
|
||||
@@ -259,7 +284,12 @@ sampled through the ONE :func:`_sample_rss` — at the run start, at every
|
||||
``phase`` transition, in every terminal, and at most once per
|
||||
``_RSS_SAMPLE_INTERVAL_S`` from the per-file progress hook, so a 27k-file
|
||||
walk costs ~600 ``/proc`` reads and not 27k — LOCKED A4), and
|
||||
``GET /api/sync/status`` reports both keys ADDITIVELY. The terminals keep
|
||||
``GET /api/sync/status`` reports both keys ADDITIVELY. Phase 194
|
||||
(task 02) extends both halves of that rule to the new stage: the
|
||||
``"scan"`` set before ``import_sources`` and the ``scan``→``import``
|
||||
flip are phase TRANSITIONS and FORCE the read, while the pre-scan's own
|
||||
per-file hook samples UNFORCED (throttled) — a 10,000-file scan costs
|
||||
one ``/proc`` read per interval, never one per file. The terminals keep
|
||||
the FINAL reading next to the final counts (the phase-64 convention), so
|
||||
the page that started the run answers the incident's question without a
|
||||
shell. A reading the platform cannot know stays ``null`` on the wire (the
|
||||
@@ -323,11 +353,33 @@ class SyncStatus:
|
||||
outside the import phase — clone/pull first, terminal states
|
||||
after); ``files_done`` / ``files_total`` carry the hook's
|
||||
done/total position and survive a terminal state (the run's last
|
||||
position is useful context next to the error).
|
||||
position is useful context next to the error). Since phase 194 the
|
||||
pair is the hook's PENDING position (phase 194: the pre-scan's
|
||||
predicted change set — ``files_total`` the files that will actually
|
||||
be re-indexed), not the walk position; ``BOR_SYNC_PENDING_COUNT=0``
|
||||
restores the walk position and every other caller (the CLI, the
|
||||
upload job) is unaffected.
|
||||
|
||||
Phase 194 (task 02) scan fields: ``scan_done`` / ``scan_total``
|
||||
carry the pre-scan's position — the 1-based file whose hash+read
|
||||
just completed over the combined importable count the scan walks
|
||||
(the importer's ``on_scan_progress`` hook). They are 0/0 before the
|
||||
scan starts and on the ``BOR_SYNC_PENDING_COUNT=0`` rollback (no
|
||||
scan, so no report), reset with the run like every other field, and
|
||||
KEPT at every terminal beside the final file counts (the phase-64
|
||||
keep-final-values convention — the scan the owner watched is still
|
||||
on the page after the run). ``current_file`` stays ``null`` through
|
||||
the phase: the scan names no file, it only counts. The degenerate
|
||||
empty knowledge base is the honest edge — the scan predicts nothing
|
||||
so no report fires and no ``progress`` call ever arrives: the phase
|
||||
stays ``"scan"`` until the next transition or the terminal (which
|
||||
clears it), and the label is the bare phase name with no count part
|
||||
(the UI's ``total === 0`` guard).
|
||||
|
||||
Phase 98 (task 01) phase fields: ``phase`` is the pipeline stage
|
||||
(null in the model-check + clone/pull prelude and in terminal
|
||||
states — the module docstring's phase machine); while in the
|
||||
states — the module docstring's phase machine, whose stages are now
|
||||
``scan`` → ``import`` → ``overview`` → ``summaries``); while in the
|
||||
``"summaries"`` phase, ``current_summary`` is the source /
|
||||
``source/folder_path`` the generator is summarizing right now and
|
||||
``summaries_done`` / ``summaries_total`` carry the generator's
|
||||
@@ -393,10 +445,20 @@ class SyncStatus:
|
||||
current_file: str | None = None
|
||||
files_done: int = 0
|
||||
files_total: int = 0
|
||||
# Phase 194 (task 02): the PRE-SCAN's position — the 1-based file
|
||||
# whose hash+read just completed and the combined importable count
|
||||
# the scan walks (the importer's on_scan_progress hook). 0/0 before
|
||||
# the scan starts (and on the BOR_SYNC_PENDING_COUNT=0 rollback);
|
||||
# reset with the run and kept at every terminal beside the final
|
||||
# file counts (the phase-64 keep-final-values convention).
|
||||
scan_done: int = 0
|
||||
scan_total: int = 0
|
||||
# Phase 98 (task 01): the phase machine — the pipeline stage the
|
||||
# run is in and, while in the folder-summary phase, the folder
|
||||
# being summarized plus the hook's done/total position.
|
||||
phase: Literal["import", "overview", "summaries"] | None = None
|
||||
# being summarized plus the hook's done/total position. Phase 194
|
||||
# (task 02) adds the "scan" stage (the pre-scan that predicts the
|
||||
# run's change set — the module docstring's phase machine).
|
||||
phase: Literal["scan", "import", "overview", "summaries"] | None = None
|
||||
current_summary: str | None = None
|
||||
summaries_done: int = 0
|
||||
summaries_total: int = 0
|
||||
@@ -433,13 +495,15 @@ _task: asyncio.Task[None] | None = None
|
||||
# starts, so a stale event can never bleed into the next run.
|
||||
_cancel_event = asyncio.Event()
|
||||
|
||||
#: Phase 187 (task 03, D3/A4): how often the PER-FILE progress hook may
|
||||
#: re-read the process memory, in seconds. That hook fires once per
|
||||
#: file, so without this guard a 27k-file walk would cost 27k
|
||||
#: ``/proc/self/status`` reads; with it the same walk costs ~600 and the
|
||||
#: operator watching the page still sees the number move (PLAN §7.4's
|
||||
#: never-stale contract). Every other sampling point (the run start, the
|
||||
#: ``phase`` transitions, the terminals) FORCES the read instead.
|
||||
#: Phase 187 (task 03, D3/A4): how often the PER-FILE progress hooks may
|
||||
#: re-read the process memory, in seconds. Those hooks fire once per
|
||||
#: file — the import's ``progress`` AND, since phase 194 (task 02), the
|
||||
#: pre-scan's ``on_scan_progress`` — so without this guard a 27k-file
|
||||
#: walk would cost 27k ``/proc/self/status`` reads; with it the same walk
|
||||
#: costs ~600 and the operator watching the page still sees the number
|
||||
#: move (PLAN §7.4's never-stale contract). Every other sampling point
|
||||
#: (the run start, the ``phase`` transitions including the ``scan`` set
|
||||
#: and its flip to ``import``, the terminals) FORCES the read instead.
|
||||
_RSS_SAMPLE_INTERVAL_S = 5.0
|
||||
|
||||
#: ``time.monotonic()`` of the last RSS read; ``-inf`` means "never", so
|
||||
@@ -503,7 +567,21 @@ def sync_status() -> dict[str, Any]:
|
||||
import is processing right now — null during the clone/pull phase
|
||||
and in terminal states; ``files_done`` / ``files_total`` carry the
|
||||
hook's position (0/0 idle) and keep their final values in the
|
||||
terminal. ``current_chunk`` / ``chunks_total`` (phase 129) are the
|
||||
terminal — the position in the pre-scan's PREDICTED change set on a
|
||||
progress caller (phase 194), the walk position under
|
||||
``BOR_SYNC_PENDING_COUNT=0``. ``scan_done`` / ``scan_total`` (phase
|
||||
194) are the PRE-SCAN's position: the 1-based file whose
|
||||
hash+compare just ran over the combined importable count the scan
|
||||
walks. They are 0/0 before the scan starts — the prelude, a run on
|
||||
the ``0`` rollback, and every run that never happened — and in a
|
||||
terminal they keep the FINAL scan values beside the final file
|
||||
counts (the phase-64 keep-final-values convention); ``current_file``
|
||||
stays null while ``phase`` is ``"scan"``, because the scan names no
|
||||
file. Both keys are ADDITIVE — placed after ``files_total``, every
|
||||
pre-existing key keeping its name, type and position. The degenerate
|
||||
empty knowledge base reports the phase without ever reporting a
|
||||
position (0/0 — the UI's ``total === 0`` guard renders no count
|
||||
part). ``current_chunk`` / ``chunks_total`` (phase 129) are the
|
||||
in-flight file's embed-chunk position (batch granularity — the
|
||||
first chunk of the in-flight batch over the file's total content
|
||||
chunks): 0/0 before the first batch, in the prelude/overview/
|
||||
@@ -551,6 +629,14 @@ def sync_status() -> dict[str, Any]:
|
||||
"current_file": _status.current_file,
|
||||
"files_done": _status.files_done,
|
||||
"files_total": _status.files_total,
|
||||
# Phase 194 (task 02): the pre-scan's position (the importer's
|
||||
# on_scan_progress hook) — 0/0 before the scan starts and on the
|
||||
# BOR_SYNC_PENDING_COUNT=0 rollback, the FINAL pair kept in every
|
||||
# terminal beside the file counts above (the keep-final-values
|
||||
# convention). Additive: every pre-existing key keeps its name,
|
||||
# type and position.
|
||||
"scan_done": _status.scan_done,
|
||||
"scan_total": _status.scan_total,
|
||||
# Phase 129 (task 03): the in-flight file's embed-chunk
|
||||
# position — the dataclass defaults make idle 0/0, and the
|
||||
# runner clears it at every point the import phase ends
|
||||
@@ -672,6 +758,14 @@ async def _run_sync() -> None:
|
||||
so a long clone no longer stalls the rest of the HTTP surface (the
|
||||
module docstring's freeze paragraph); the run's own sequencing is
|
||||
byte-for-byte the pipeline above.
|
||||
|
||||
Phase 194 (task 02): the phase set before ``import_sources`` is
|
||||
``"scan"`` (not ``"import"``), because that call begins with the
|
||||
off-loop pre-scan that predicts the change set; the FIRST per-file
|
||||
``progress`` call flips it to ``"import"`` inside ``_hook``. Both
|
||||
transitions force the memory sample, the scan's own per-file hook
|
||||
does not (LOCKED A4), and ``scan_done`` / ``scan_total`` reset with
|
||||
the run and survive every terminal beside the file counts.
|
||||
"""
|
||||
_status.state = "running"
|
||||
_status.started_at = datetime.now(UTC)
|
||||
@@ -683,6 +777,11 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None
|
||||
_status.files_done = 0
|
||||
_status.files_total = 0
|
||||
# Phase 194 (task 02): the pre-scan's position resets with the run —
|
||||
# 0/0 until the scan reports its first file (and 0/0 for the whole
|
||||
# run on the BOR_SYNC_PENDING_COUNT=0 rollback, which scans nothing).
|
||||
_status.scan_done = 0
|
||||
_status.scan_total = 0
|
||||
# Phase 98 (task 01): the phase fields reset with the run — no
|
||||
# phase until the import starts (the model-check + clone/pull
|
||||
# prelude reports null), no folder until the summary span starts.
|
||||
@@ -834,6 +933,17 @@ async def _run_sync() -> None:
|
||||
# run-start reset alone cannot cover it).
|
||||
_status.images_done = 0
|
||||
_status.images_total = 0
|
||||
# Phase 194 (task 02): the scan→import FLIP. The phase is
|
||||
# set to "scan" immediately before import_sources, and the
|
||||
# first file the walk actually reaches is where the running
|
||||
# label stops being a scan — no extra callback reaches the
|
||||
# importer, the flip lives in this existing per-file hook.
|
||||
if _status.phase == "scan":
|
||||
_status.phase = "import"
|
||||
# A phase TRANSITION is a forced sampling point (phase
|
||||
# 187), so this one read is not the throttled per-file
|
||||
# one the tail of this closure does.
|
||||
_sample_rss(force=True)
|
||||
# Phase 187 (task 03): the run's memory, sampled THROTTLED
|
||||
# (unforced) — this hook fires once per file, so a 27k-file
|
||||
# walk reads /proc ~600 times instead of 27k (LOCKED A4)
|
||||
@@ -865,10 +975,30 @@ async def _run_sync() -> None:
|
||||
_status.images_done = done
|
||||
_status.images_total = total
|
||||
|
||||
# Phase 98 (task 01): the phase machine — the import phase
|
||||
# starts NOW (the model-check + clone/pull prelude above
|
||||
# reported ``phase: null``).
|
||||
_status.phase = "import"
|
||||
# Phase 194 (task 02): the PRE-SCAN's position — the importer's
|
||||
# ``on_scan_progress`` hook, fired once per importable file while
|
||||
# the run's change set is still being PREDICTED (before the walk),
|
||||
# with (position_1based, scan_total). It runs on the scan's
|
||||
# WORKER thread, so this closure only assigns fields on the
|
||||
# module's own status object — the house hook contract that makes
|
||||
# each call a single GIL-atomic store (phase 179). The memory
|
||||
# sample is UNFORCED here: this fires once per file, exactly like
|
||||
# ``_hook`` above, so it costs one /proc read per interval and
|
||||
# never one per file (LOCKED A4).
|
||||
def _scan_hook(scan: int, total: int) -> None:
|
||||
_status.scan_done = scan
|
||||
_status.scan_total = total
|
||||
_sample_rss()
|
||||
|
||||
# Phase 98 (task 01) + phase 194 (task 02): the phase machine —
|
||||
# the SCAN phase starts NOW (the model-check + clone/pull prelude
|
||||
# above reported ``phase: null``). What runs first inside
|
||||
# ``import_sources`` is the pre-scan that predicts the change set,
|
||||
# and the label the owner reads through it is "Scanning…"
|
||||
# (phase 193's "import" set moved to the first ``_hook`` call
|
||||
# above — the flip), so the count the page shows is never a scan
|
||||
# wearing the work's shape.
|
||||
_status.phase = "scan"
|
||||
# Phase 187 (task 03): the phase transitions are the forced
|
||||
# sampling points — each stage of the run begins with its own
|
||||
# reading on the status (the per-file hook throttles).
|
||||
@@ -885,6 +1015,7 @@ async def _run_sync() -> None:
|
||||
progress=_hook,
|
||||
on_chunk_progress=_chunk_hook,
|
||||
on_image_progress=_image_hook, # phase 152 (task 03)
|
||||
on_scan_progress=_scan_hook, # phase 194 (task 02)
|
||||
ignore_by_root=ignore_by_root,
|
||||
include_hidden_by_root=include_hidden_by_root,
|
||||
doc_dates_by_root=doc_dates_by_root,
|
||||
@@ -1034,6 +1165,10 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # phase 64: keep the final counts
|
||||
_status.phase = None # phase 98: keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 194 (task 02): the scan position is NOT cleared here —
|
||||
# like the file counts it keeps its final values beside them
|
||||
# (the keep-final-values convention), so the page reloaded after
|
||||
# the run still says how big the predicted change set was.
|
||||
# Phase 129 (task 03): the chunk position is a sub-position of
|
||||
# a file that's done — it clears in the terminal (unlike the
|
||||
# FILE counts, which keep their final values for the result
|
||||
@@ -1091,6 +1226,10 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # keep the final counts
|
||||
_status.phase = None # keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 194 (task 02): the scan position survives the cancel
|
||||
# too — where a cancel landed DURING the pre-scan it is the
|
||||
# run's only number, and the log line above keeps the file
|
||||
# pair beside it (the keep-final-values convention).
|
||||
# Phase 129 (task 03): the chunk position is a sub-position
|
||||
# of a file that's done — it clears in EVERY terminal, like
|
||||
# success and failed (the convention split, pinned).
|
||||
@@ -1138,6 +1277,9 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # phase 64: keep the final counts
|
||||
_status.phase = None # phase 98: keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 194 (task 02): the scan position is kept here as well —
|
||||
# a run that died mid-scan keeps the position it reached beside
|
||||
# the final file counts (the convention, every terminal).
|
||||
# Phase 129 (task 03): the chunk position clears in the
|
||||
# failed terminal too (the convention split — the FILE counts
|
||||
# keep their final values next to the error, the sub-position
|
||||
|
||||
@@ -524,6 +524,22 @@ class Settings(BaseSettings):
|
||||
#: mid-prune leaves a PREFIX pruned, and the next sync converges
|
||||
#: because ``seen`` is recomputed from the filesystem every run.
|
||||
sync_prune_batch: int = 500
|
||||
#: Predict the run's change set BEFORE the walk, so the sync's running
|
||||
#: count is the count of the WORK and not of the walk (phase 194,
|
||||
#: ``BOR_SYNC_PENDING_COUNT``). ``1`` (default): ``import_sources``
|
||||
#: pre-scans the importable files (one streamed sha256 + one scalar
|
||||
#: ``documents.file_hash`` read each, off the event loop) and the
|
||||
#: per-file ``progress`` hook then reports ``done``/``total`` as the
|
||||
#: position in that PENDING set — the owner's "1216 remaining" fix.
|
||||
#: ``0`` = the exact phase-193 behaviour (the stats-only pre-walk and
|
||||
#: ``done = summary.files``, i.e. "every file left to WALK") — the
|
||||
#: operator's no-redeploy rollback if the extra read cost ever bites.
|
||||
#: A negative value is a typo — the validator fails startup loudly
|
||||
#: (the ``agent_max_rounds`` pattern). Read per run (the importer reads
|
||||
#: ``llm.settings`` once per sync, so a saved override applies to the
|
||||
#: NEXT sync) and only by a caller that passes ``progress`` — the CLI
|
||||
#: and the upload job pay nothing.
|
||||
sync_pending_count: int = 1
|
||||
|
||||
# --- Image documents (phase 122: standalone images as documents) ---
|
||||
#: Master switch for image-document indexing (phase 122,
|
||||
@@ -930,6 +946,20 @@ class Settings(BaseSettings):
|
||||
)
|
||||
return v
|
||||
|
||||
@field_validator("sync_pending_count")
|
||||
@classmethod
|
||||
def _sync_pending_count_non_negative(cls, v: int) -> int:
|
||||
"""``0`` is the phase-193 rollback (count the walk, not the work) —
|
||||
a negative value is a typo that no branch of the importer can mean,
|
||||
so it fails startup naming the field (the ``agent_max_rounds``
|
||||
pattern, phase 194)."""
|
||||
if v < 0:
|
||||
raise ValueError(
|
||||
"sync_pending_count must be >= 0 (0 = count the whole walk, "
|
||||
"the pre-phase-194 behaviour)"
|
||||
)
|
||||
return v
|
||||
|
||||
@field_validator("chat_image_max_mb")
|
||||
@classmethod
|
||||
def _chat_image_max_mb_positive(cls, v: int) -> int:
|
||||
|
||||
@@ -304,6 +304,9 @@ CATALOG: dict[str, SettingMeta] = {
|
||||
"sync_prune_batch": SettingMeta(
|
||||
"sync_prune_batch", "Import & sources", "Sync prune batch"
|
||||
),
|
||||
"sync_pending_count": SettingMeta(
|
||||
"sync_pending_count", "Import & sources", "Sync pending count"
|
||||
),
|
||||
# --- Images ---
|
||||
"images": SettingMeta("images", "Images", "Images"),
|
||||
"image_extensions": SettingMeta(
|
||||
|
||||
+287
-30
@@ -332,14 +332,27 @@ convention, phase 133).
|
||||
(phase 64, task 01) reporting the file being processed right now, an
|
||||
optional ``on_chunk_progress`` callback (phase 129, task 02)
|
||||
reporting the in-flight file's embedding-chunk position at batch
|
||||
granularity, and an optional ``on_image_progress`` callback (phase
|
||||
152, task 02) reporting the in-flight PDF's vision-image position —
|
||||
granularity, an optional ``on_image_progress`` callback (phase 152,
|
||||
task 02) reporting the in-flight PDF's vision-image position —
|
||||
immediately before each of the PDF's vision describe calls, with the
|
||||
image's 1-based position in the file's ONE continuous vision counter
|
||||
and the span's count (see the function docstring).
|
||||
and the span's count — and an optional ``on_scan_progress`` callback
|
||||
(phase 194, task 01) reporting the PRE-SCAN's position while the run's
|
||||
change set is predicted. The ``progress`` pair is the pending position
|
||||
(``total`` = the size of the predicted change set, ``done`` = the 1-based
|
||||
position in it) rather than the walk position, which is what makes the
|
||||
Sources page's ``N remaining`` count the WORK and not the WALK: a
|
||||
~10,000-file sync that changes 10 files says "10 remaining", not
|
||||
"1216 remaining" (the owner report of 2026-10-19). ``0`` in
|
||||
``BOR_SYNC_PENDING_COUNT`` restores the phase-193 walk position exactly.
|
||||
All four callbacks follow the same "only when provided" rule — a caller
|
||||
that passes nothing pays the pass, the read and the cost of none of them,
|
||||
and a caller that passes no ``progress`` runs no pre-scan at all (the
|
||||
CLI, the upload job: byte-identical). See the function docstring.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
@@ -753,6 +766,125 @@ def _doc_row_stmt(source: str, rel: str) -> Select[tuple[Document]]:
|
||||
)
|
||||
|
||||
|
||||
def _pending_hash_stmt(source: str, rel: str) -> Select[tuple[str | None]]:
|
||||
"""The pre-scan's ONE-column metadata read (phase 194, task 01).
|
||||
|
||||
``select(Document.file_hash)`` is a SCALAR projection, not an entity
|
||||
load: no ``Document`` is ever constructed, nothing lands in the run's
|
||||
``identity_map``, and the pre-scan therefore retains nothing — which is
|
||||
why it needs no ``defer``/``load_only`` (there is no row to defer) and
|
||||
why it does NOT touch :func:`_doc_row_stmt`, which stays the importer's
|
||||
ONLY row load (phase 185). The predicate is the unique
|
||||
``(source, path)`` constraint (``app/models.py``, the same lookup the
|
||||
fast path uses), so the read costs one index probe per file — the
|
||||
``file_hash`` column itself stays unindexed by design.
|
||||
"""
|
||||
return select(Document.file_hash).where(
|
||||
Document.source == source, Document.path == rel
|
||||
)
|
||||
|
||||
|
||||
def _prescan_pending(
|
||||
sources: Sequence[Path],
|
||||
extensions: frozenset[str],
|
||||
image_extensions: frozenset[str],
|
||||
ignore_by_root: dict[str, list[str]] | None,
|
||||
include_hidden_by_root: dict[str, bool] | None,
|
||||
on_scan_progress: Callable[[int, int], None] | None,
|
||||
session: Session | None = None,
|
||||
) -> tuple[int, dict[tuple[str, str], int]]:
|
||||
"""Predict the run's change set BEFORE the walk (phase 194, D1).
|
||||
|
||||
Returns ``(scan_total, pending)``: the combined count of importable
|
||||
files across *sources* — the SAME number the phase-64 stats-only pre-walk
|
||||
computed, under the SAME ``iter_importable_files`` rules (the caller
|
||||
passes the exact *extensions* / *image_extensions* / per-root ``ignore``
|
||||
and ``include_hidden`` values the processing loop uses, so a file the
|
||||
loop cannot see is never predicted) — and ``pending``, the map of
|
||||
``(source, rel)`` to that file's 1-based position in the predicted change
|
||||
set, walk order.
|
||||
|
||||
Per file: :func:`_file_hash` (the streamed 1 MiB-chunk sha256 of phase
|
||||
147 — the raw bytes, the sync's single change signal) plus ONE
|
||||
:func:`_pending_hash_stmt` scalar read, and the file is pending when the
|
||||
stored hash is ``NULL`` (no row yet, or a pre-phase-147 row) or differs
|
||||
from the digest — the SAME predicate the fast path applies at processing
|
||||
time, so a pre-phase NULL row is predicted changed AND really re-indexed
|
||||
and stamped by the loop: the prediction never lies about the stored-hash
|
||||
contract. A file the digest cannot reach (deleted between the walk and
|
||||
the hash) is predicted changed too: the loop's own walk no longer sees
|
||||
it, so the count can hold one short, which is honest — the terminal
|
||||
counts line is the exact surface either way.
|
||||
|
||||
``session`` follows the importer's own convention (tests pass one, a
|
||||
private one is opened and closed otherwise). Deliberately NOT the RUN's
|
||||
session: this function executes inside an :func:`asyncio.to_thread`
|
||||
worker (see :func:`import_sources`), and a ``Session`` is not
|
||||
thread-safe — a cancel at the span's await lets this worker run to
|
||||
completion while the run has already moved on to its terminal and closed
|
||||
its own session. Reading through a session this span owns is what makes
|
||||
that edge safe; the projection loads no entity, so even this session
|
||||
keeps nothing.
|
||||
|
||||
*on_scan_progress* is called — only when provided — once per file with
|
||||
``(position_1based, scan_total)``, AFTER that file's hash+read (so the
|
||||
position counts files actually scanned and reaches the total exactly when
|
||||
the scan does). It runs on the WORKER thread: the house hook contract is
|
||||
that these hooks only assign fields on the caller's own status object,
|
||||
which makes each call a single GIL-atomic store, and there is no
|
||||
try/except around it for the same reason as the three sibling hooks.
|
||||
"""
|
||||
owns_session = session is None
|
||||
if session is None:
|
||||
session = SessionLocal()
|
||||
try:
|
||||
# (a) the per-root file lists + the combined length, so the scan's
|
||||
# denominator is known before the first callback. Directory stats
|
||||
# only — no file read, no database statement (the phase-64 shape).
|
||||
per_root: list[tuple[Path, list[Path]]] = [
|
||||
(
|
||||
root,
|
||||
iter_importable_files(
|
||||
root,
|
||||
extensions,
|
||||
ignore=_ignore_for_root(root, ignore_by_root),
|
||||
include_hidden=_include_hidden_for_root(
|
||||
root, include_hidden_by_root
|
||||
),
|
||||
image_extensions=image_extensions,
|
||||
),
|
||||
)
|
||||
for root in sources
|
||||
]
|
||||
scan_total = sum(len(files) for _, files in per_root)
|
||||
pending: dict[tuple[str, str], int] = {}
|
||||
position = 0
|
||||
for root, files in per_root:
|
||||
source = root.name
|
||||
for path in files:
|
||||
rel = path.relative_to(root).as_posix()
|
||||
try:
|
||||
digest = _file_hash(path)
|
||||
except OSError as e: # vanished mid-scan — predict the change
|
||||
logger.warning(
|
||||
"import: pre-scan unreadable source=%s path=%s — %s",
|
||||
source,
|
||||
rel,
|
||||
e,
|
||||
)
|
||||
digest = ""
|
||||
stored = session.scalar(_pending_hash_stmt(source, rel))
|
||||
if stored is None or stored != digest:
|
||||
pending[(source, rel)] = len(pending) + 1
|
||||
position += 1
|
||||
if on_scan_progress is not None:
|
||||
on_scan_progress(position, scan_total)
|
||||
return scan_total, pending
|
||||
finally:
|
||||
if owns_session:
|
||||
session.close()
|
||||
|
||||
|
||||
async def import_sources(
|
||||
sources: list[Path],
|
||||
llm: Embedder,
|
||||
@@ -763,6 +895,7 @@ async def import_sources(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
on_image_progress: Callable[[int, int], None] | None = None,
|
||||
on_scan_progress: Callable[[int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
@@ -776,15 +909,53 @@ async def import_sources(
|
||||
``progress`` (phase 64, task 01) is an optional per-file hook called
|
||||
once per importable file, immediately before that file's
|
||||
``_index_file`` — with ``(source, rel_posix_path, done, total)``: the
|
||||
same POSIX *rel* the document rows use, ``done`` = the 1-based index of
|
||||
the current file **across all sources**, and ``total`` = the combined
|
||||
pre-walk count of importable files across all *sources* roots. The
|
||||
pre-walk (same extension/exclusion rules, directory stats only, no file
|
||||
reads) happens **only when *progress* is provided**: callers passing
|
||||
nothing pay no extra walk and behave exactly as before. Under
|
||||
``limit``, the hook still fires per processed file only — ``done`` never
|
||||
exceeds the limit, but ``total`` stays the full pre-walk count (an
|
||||
incomplete walk must not misreport the denominator).
|
||||
same POSIX *rel* the document rows use, plus the position pair the UI's
|
||||
running count consumes. The pair's MEANING is the phase-194 switch
|
||||
``llm.settings.sync_pending_count`` (read ONCE before the source loop —
|
||||
see the pre-scan paragraph below):
|
||||
|
||||
* switch truthy (the default): ``total`` = the size of the pre-scan's
|
||||
PENDING set (the files predicted to be re-indexed) and ``done`` = the
|
||||
1-based position in THAT set. The position advances for every file in
|
||||
the pending set — whether or not the file turns out changed at
|
||||
processing time — so count and denominator stay on the SAME set and
|
||||
the arithmetic never runs away; a file NOT in the set does not advance
|
||||
it, which is what keeps the label holding at ``K remaining`` while the
|
||||
walk moves over the unchanged ``N − K``.
|
||||
* switch ``0``: the exact phase-193 walk position — ``done`` = the
|
||||
1-based index of the current file **across all sources** and ``total``
|
||||
= the combined pre-walk count of importable files (the full rollback).
|
||||
|
||||
In both states the count-only pass happens **only when *progress* is
|
||||
provided**: callers passing nothing pay no extra pass and behave exactly
|
||||
as before (the CLI, the upload job). Under ``limit``, the hook still
|
||||
fires per processed file only — ``done`` never exceeds the limit, but
|
||||
``total`` stays the full count (an incomplete walk must not misreport
|
||||
the denominator).
|
||||
|
||||
The pair is a PREDICTION, and its drift edges are bounded (phase 194
|
||||
A1): a file the scan predicted and that reverted to unchanged before the
|
||||
loop reached it still advances the count (it was counted in ``total``,
|
||||
so the count still reaches 0); a file changed AFTER the scan is indexed
|
||||
for real but advances nothing (``total`` never counted it), and the
|
||||
UI's ``total - done > 0`` guard hides that excess — never a negative and
|
||||
never a stale ``0 remaining``. The terminal ``counts_line`` ("N added ·
|
||||
M updated …") stays the exact surface, untouched.
|
||||
|
||||
``on_scan_progress`` (phase 194, task 01) is an optional SCAN-position
|
||||
hook for the pre-scan itself: while the pending set is being predicted it
|
||||
is called — once per importable file, after that file's hash+read — with
|
||||
``(position_1based, scan_total)``, the file's 1-based position in the
|
||||
scan and the combined importable count across all *sources* roots (the
|
||||
number the phase-64 stats pre-walk used to compute). It carries counts
|
||||
only, never the file identity (the sibling convention), and it runs on
|
||||
the span's WORKER thread — the house rule that these hooks only assign
|
||||
fields is what makes each call a single GIL-atomic store. Same "only
|
||||
when provided" rule as ``progress``, ``on_chunk_progress`` and
|
||||
``on_image_progress``, with one addition: the pre-scan it reports on is
|
||||
gated on *progress* too, so a caller passing ``on_scan_progress`` alone
|
||||
(``progress=None``) runs no scan, fires no callback and stays
|
||||
byte-identical — the scan exists only to serve the running counts.
|
||||
|
||||
``on_chunk_progress`` (phase 129, task 02) is an optional
|
||||
chunk-position hook for the file currently being indexed: while that
|
||||
@@ -923,6 +1094,45 @@ async def import_sources(
|
||||
(``0`` = never — the exact pre-phase behaviour, the operator's
|
||||
no-redeploy rollback). The cadence changes no count, no log line and no
|
||||
committed byte: see the safety argument at the reset site.
|
||||
|
||||
The pre-scan (phase 194, task 01): with the hook on and
|
||||
``llm.settings.sync_pending_count`` truthy, the count-only pass is no
|
||||
longer a stats walk but :func:`_prescan_pending` — every importable file
|
||||
hashed once and compared once against its stored ``file_hash``, giving
|
||||
the PENDING set the ``progress`` pair counts. Three costs the phase
|
||||
accepted, deliberately:
|
||||
|
||||
* **off the loop** (the phase-179 rule): the scan runs in ONE
|
||||
:func:`asyncio.to_thread` span. Ten thousand streamed hashes plus ten
|
||||
thousand one-column SELECTs inline would park the single uvicorn loop
|
||||
for seconds — the exact defect class
|
||||
``tests/integration/test_sync_git_freeze.py`` pins for the git clone,
|
||||
and the ``GET /api/sync/status`` poll the UI runs every 2 s is the
|
||||
victim. A cancel registered at the span's await interrupts the RUN
|
||||
(phase 134: it lands the ``cancelled`` terminal), while the worker
|
||||
itself runs to completion — threads are not interruptible, which is
|
||||
also why the span reads through a session of its own instead of the
|
||||
run's (see :func:`_prescan_pending`).
|
||||
* **one extra read + hash + scalar SELECT per importable file per run**
|
||||
(A3): the second read of the same file is served from the page cache,
|
||||
and the scalar read is a unique ``(source, path)`` index probe.
|
||||
``BOR_SYNC_PENDING_COUNT=0`` buys the whole cost back and restores the
|
||||
phase-193 walk-position semantics byte-for-byte; a negative value
|
||||
refuses startup naming the field.
|
||||
* **the file is hashed twice**, here and again in ``_index_file``. That
|
||||
is deliberate: the pre-scan's digest is the PREDICTION, the loop's is
|
||||
the TRUTH — a file edited between the two is still detected and
|
||||
re-indexed. Dropping the second hash would make the count a claim
|
||||
about bytes nobody verified.
|
||||
|
||||
Everything else in the run is untouched by the scan: ``summary.files``
|
||||
(the ``import: summary … files=`` count of every file the loop
|
||||
processed), ``seen``, the prune post-pass, the date refresh and every log
|
||||
line keep their exact shape — the pre-scan builds ``pending``, never
|
||||
``seen``, and the loop keeps building ``seen`` from disk exactly as
|
||||
before. The degenerate empty knowledge base is the phase-64 shape: scan
|
||||
total ``0``, no callback fires, the pending set is empty and ``total``
|
||||
is ``0`` (the UI's ``total === 0`` guard then renders no count part).
|
||||
"""
|
||||
if limit is not None and limit <= 0:
|
||||
raise ValueError("limit must be >= 1")
|
||||
@@ -937,30 +1147,68 @@ async def import_sources(
|
||||
# reads ``llm.settings`` per RUN, never per file). ``0`` disables the
|
||||
# release entirely.
|
||||
reset_every = llm.settings.sync_session_reset_files
|
||||
# Phase 194 (task 01): the pending-count switch, read ONCE here — before
|
||||
# the source loop — so a settings overlay saved mid-run cannot change the
|
||||
# MEANING of the hook's ``done``/``total`` pair between two files of the
|
||||
# same walk (the phase-184 read-once discipline). ``0`` = the exact
|
||||
# phase-193 walk-position semantics, the operator's no-redeploy rollback.
|
||||
count_pending = bool(llm.settings.sync_pending_count)
|
||||
owns_session = session is None
|
||||
if session is None:
|
||||
session = SessionLocal()
|
||||
seen: set[tuple[str, str]] = set()
|
||||
source_names: set[str] = set()
|
||||
# phase 64 (task 01): the hook's combined denominator, walked with the
|
||||
# exact same rules as the processing loop below (directory stats only,
|
||||
# no file reads). Skipped entirely for ``progress=None`` callers — no
|
||||
# extra pass, byte-identical behaviour and cost.
|
||||
# phase 64 (task 01): the hook's denominator. phase 194 (task 01): what
|
||||
# the number MEANS follows ``count_pending`` — the size of the predicted
|
||||
# change set, or the count of every importable file. Either way the pass
|
||||
# that computes it is skipped entirely for ``progress=None`` callers (no
|
||||
# extra cost, byte-identical behaviour) and it is INSIDE the try so the
|
||||
# session is closed on every exit path.
|
||||
total = 0
|
||||
if progress is not None:
|
||||
for root in sources:
|
||||
total += len(
|
||||
iter_importable_files(
|
||||
root,
|
||||
llm.settings.import_extension_set,
|
||||
ignore=_ignore_for_root(root, ignore_by_root),
|
||||
include_hidden=_include_hidden_for_root(
|
||||
root, include_hidden_by_root
|
||||
),
|
||||
image_extensions=image_exts,
|
||||
)
|
||||
)
|
||||
# (source, rel) -> 1-based position in the PENDING set; ``None`` = the
|
||||
# phase-193 walk-position semantics (the switch ``0`` branch below).
|
||||
pending: dict[tuple[str, str], int] | None = None
|
||||
# the position the hook last reported: a file the scan did not predict
|
||||
# holds it, which keeps ``total - done`` honest while the walk moves over
|
||||
# the unchanged files.
|
||||
pending_done = 0
|
||||
try:
|
||||
if progress is not None:
|
||||
if count_pending:
|
||||
# THE PRE-SCAN (phase 194, D1) — it REPLACES the stats-only
|
||||
# pre-walk, so there is no third walk. ONE off-loop span
|
||||
# (the phase-179 rule: a 10,000-file hash+SELECT pass inline
|
||||
# would park the single uvicorn loop and freeze the status
|
||||
# poll the UI runs every 2 s). The scan total is the scan
|
||||
# hook's own business; the run's denominator is the pending
|
||||
# set the scan returns.
|
||||
_scan_total, pending = await asyncio.to_thread(
|
||||
_prescan_pending,
|
||||
sources,
|
||||
llm.settings.import_extension_set,
|
||||
image_exts,
|
||||
ignore_by_root,
|
||||
include_hidden_by_root,
|
||||
on_scan_progress,
|
||||
)
|
||||
total = len(pending)
|
||||
else:
|
||||
# ``BOR_SYNC_PENDING_COUNT=0`` — the phase-193 rollback,
|
||||
# byte-for-byte: the hook's combined denominator, walked with
|
||||
# the exact same rules as the processing loop below (directory
|
||||
# stats only, no file reads).
|
||||
for root in sources:
|
||||
total += len(
|
||||
iter_importable_files(
|
||||
root,
|
||||
llm.settings.import_extension_set,
|
||||
ignore=_ignore_for_root(root, ignore_by_root),
|
||||
include_hidden=_include_hidden_for_root(
|
||||
root, include_hidden_by_root
|
||||
),
|
||||
image_extensions=image_exts,
|
||||
)
|
||||
)
|
||||
for root in sources:
|
||||
if not root.is_dir():
|
||||
logger.warning("import: source dir not found, skipping: %s", root)
|
||||
@@ -1036,7 +1284,16 @@ async def import_sources(
|
||||
# file that then errors or turns out unchanged was
|
||||
# already the "current file". No try/except around the
|
||||
# call: the hooks in this repo only assign fields.
|
||||
progress(source, rel, summary.files, total)
|
||||
# phase 194 (task 01): the pair's meaning follows the
|
||||
# switch read before the loop — the position in the
|
||||
# pending set, or the pre-phase-194 walk position.
|
||||
if pending is not None:
|
||||
predicted = pending.get((source, rel))
|
||||
if predicted is not None:
|
||||
pending_done = predicted
|
||||
progress(source, rel, pending_done, total)
|
||||
else:
|
||||
progress(source, rel, summary.files, total)
|
||||
try:
|
||||
await _index_file(
|
||||
session, source=source, rel=rel, full_path=path, llm=llm,
|
||||
|
||||
Reference in New Issue
Block a user