phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
+287
-30
@@ -332,14 +332,27 @@ convention, phase 133).
|
||||
(phase 64, task 01) reporting the file being processed right now, an
|
||||
optional ``on_chunk_progress`` callback (phase 129, task 02)
|
||||
reporting the in-flight file's embedding-chunk position at batch
|
||||
granularity, and an optional ``on_image_progress`` callback (phase
|
||||
152, task 02) reporting the in-flight PDF's vision-image position —
|
||||
granularity, an optional ``on_image_progress`` callback (phase 152,
|
||||
task 02) reporting the in-flight PDF's vision-image position —
|
||||
immediately before each of the PDF's vision describe calls, with the
|
||||
image's 1-based position in the file's ONE continuous vision counter
|
||||
and the span's count (see the function docstring).
|
||||
and the span's count — and an optional ``on_scan_progress`` callback
|
||||
(phase 194, task 01) reporting the PRE-SCAN's position while the run's
|
||||
change set is predicted. The ``progress`` pair is the pending position
|
||||
(``total`` = the size of the predicted change set, ``done`` = the 1-based
|
||||
position in it) rather than the walk position, which is what makes the
|
||||
Sources page's ``N remaining`` count the WORK and not the WALK: a
|
||||
~10,000-file sync that changes 10 files says "10 remaining", not
|
||||
"1216 remaining" (the owner report of 2026-10-19). ``0`` in
|
||||
``BOR_SYNC_PENDING_COUNT`` restores the phase-193 walk position exactly.
|
||||
All four callbacks follow the same "only when provided" rule — a caller
|
||||
that passes nothing pays the pass, the read and the cost of none of them,
|
||||
and a caller that passes no ``progress`` runs no pre-scan at all (the
|
||||
CLI, the upload job: byte-identical). See the function docstring.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
@@ -753,6 +766,125 @@ def _doc_row_stmt(source: str, rel: str) -> Select[tuple[Document]]:
|
||||
)
|
||||
|
||||
|
||||
def _pending_hash_stmt(source: str, rel: str) -> Select[tuple[str | None]]:
|
||||
"""The pre-scan's ONE-column metadata read (phase 194, task 01).
|
||||
|
||||
``select(Document.file_hash)`` is a SCALAR projection, not an entity
|
||||
load: no ``Document`` is ever constructed, nothing lands in the run's
|
||||
``identity_map``, and the pre-scan therefore retains nothing — which is
|
||||
why it needs no ``defer``/``load_only`` (there is no row to defer) and
|
||||
why it does NOT touch :func:`_doc_row_stmt`, which stays the importer's
|
||||
ONLY row load (phase 185). The predicate is the unique
|
||||
``(source, path)`` constraint (``app/models.py``, the same lookup the
|
||||
fast path uses), so the read costs one index probe per file — the
|
||||
``file_hash`` column itself stays unindexed by design.
|
||||
"""
|
||||
return select(Document.file_hash).where(
|
||||
Document.source == source, Document.path == rel
|
||||
)
|
||||
|
||||
|
||||
def _prescan_pending(
|
||||
sources: Sequence[Path],
|
||||
extensions: frozenset[str],
|
||||
image_extensions: frozenset[str],
|
||||
ignore_by_root: dict[str, list[str]] | None,
|
||||
include_hidden_by_root: dict[str, bool] | None,
|
||||
on_scan_progress: Callable[[int, int], None] | None,
|
||||
session: Session | None = None,
|
||||
) -> tuple[int, dict[tuple[str, str], int]]:
|
||||
"""Predict the run's change set BEFORE the walk (phase 194, D1).
|
||||
|
||||
Returns ``(scan_total, pending)``: the combined count of importable
|
||||
files across *sources* — the SAME number the phase-64 stats-only pre-walk
|
||||
computed, under the SAME ``iter_importable_files`` rules (the caller
|
||||
passes the exact *extensions* / *image_extensions* / per-root ``ignore``
|
||||
and ``include_hidden`` values the processing loop uses, so a file the
|
||||
loop cannot see is never predicted) — and ``pending``, the map of
|
||||
``(source, rel)`` to that file's 1-based position in the predicted change
|
||||
set, walk order.
|
||||
|
||||
Per file: :func:`_file_hash` (the streamed 1 MiB-chunk sha256 of phase
|
||||
147 — the raw bytes, the sync's single change signal) plus ONE
|
||||
:func:`_pending_hash_stmt` scalar read, and the file is pending when the
|
||||
stored hash is ``NULL`` (no row yet, or a pre-phase-147 row) or differs
|
||||
from the digest — the SAME predicate the fast path applies at processing
|
||||
time, so a pre-phase NULL row is predicted changed AND really re-indexed
|
||||
and stamped by the loop: the prediction never lies about the stored-hash
|
||||
contract. A file the digest cannot reach (deleted between the walk and
|
||||
the hash) is predicted changed too: the loop's own walk no longer sees
|
||||
it, so the count can hold one short, which is honest — the terminal
|
||||
counts line is the exact surface either way.
|
||||
|
||||
``session`` follows the importer's own convention (tests pass one, a
|
||||
private one is opened and closed otherwise). Deliberately NOT the RUN's
|
||||
session: this function executes inside an :func:`asyncio.to_thread`
|
||||
worker (see :func:`import_sources`), and a ``Session`` is not
|
||||
thread-safe — a cancel at the span's await lets this worker run to
|
||||
completion while the run has already moved on to its terminal and closed
|
||||
its own session. Reading through a session this span owns is what makes
|
||||
that edge safe; the projection loads no entity, so even this session
|
||||
keeps nothing.
|
||||
|
||||
*on_scan_progress* is called — only when provided — once per file with
|
||||
``(position_1based, scan_total)``, AFTER that file's hash+read (so the
|
||||
position counts files actually scanned and reaches the total exactly when
|
||||
the scan does). It runs on the WORKER thread: the house hook contract is
|
||||
that these hooks only assign fields on the caller's own status object,
|
||||
which makes each call a single GIL-atomic store, and there is no
|
||||
try/except around it for the same reason as the three sibling hooks.
|
||||
"""
|
||||
owns_session = session is None
|
||||
if session is None:
|
||||
session = SessionLocal()
|
||||
try:
|
||||
# (a) the per-root file lists + the combined length, so the scan's
|
||||
# denominator is known before the first callback. Directory stats
|
||||
# only — no file read, no database statement (the phase-64 shape).
|
||||
per_root: list[tuple[Path, list[Path]]] = [
|
||||
(
|
||||
root,
|
||||
iter_importable_files(
|
||||
root,
|
||||
extensions,
|
||||
ignore=_ignore_for_root(root, ignore_by_root),
|
||||
include_hidden=_include_hidden_for_root(
|
||||
root, include_hidden_by_root
|
||||
),
|
||||
image_extensions=image_extensions,
|
||||
),
|
||||
)
|
||||
for root in sources
|
||||
]
|
||||
scan_total = sum(len(files) for _, files in per_root)
|
||||
pending: dict[tuple[str, str], int] = {}
|
||||
position = 0
|
||||
for root, files in per_root:
|
||||
source = root.name
|
||||
for path in files:
|
||||
rel = path.relative_to(root).as_posix()
|
||||
try:
|
||||
digest = _file_hash(path)
|
||||
except OSError as e: # vanished mid-scan — predict the change
|
||||
logger.warning(
|
||||
"import: pre-scan unreadable source=%s path=%s — %s",
|
||||
source,
|
||||
rel,
|
||||
e,
|
||||
)
|
||||
digest = ""
|
||||
stored = session.scalar(_pending_hash_stmt(source, rel))
|
||||
if stored is None or stored != digest:
|
||||
pending[(source, rel)] = len(pending) + 1
|
||||
position += 1
|
||||
if on_scan_progress is not None:
|
||||
on_scan_progress(position, scan_total)
|
||||
return scan_total, pending
|
||||
finally:
|
||||
if owns_session:
|
||||
session.close()
|
||||
|
||||
|
||||
async def import_sources(
|
||||
sources: list[Path],
|
||||
llm: Embedder,
|
||||
@@ -763,6 +895,7 @@ async def import_sources(
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
on_image_progress: Callable[[int, int], None] | None = None,
|
||||
on_scan_progress: Callable[[int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
@@ -776,15 +909,53 @@ async def import_sources(
|
||||
``progress`` (phase 64, task 01) is an optional per-file hook called
|
||||
once per importable file, immediately before that file's
|
||||
``_index_file`` — with ``(source, rel_posix_path, done, total)``: the
|
||||
same POSIX *rel* the document rows use, ``done`` = the 1-based index of
|
||||
the current file **across all sources**, and ``total`` = the combined
|
||||
pre-walk count of importable files across all *sources* roots. The
|
||||
pre-walk (same extension/exclusion rules, directory stats only, no file
|
||||
reads) happens **only when *progress* is provided**: callers passing
|
||||
nothing pay no extra walk and behave exactly as before. Under
|
||||
``limit``, the hook still fires per processed file only — ``done`` never
|
||||
exceeds the limit, but ``total`` stays the full pre-walk count (an
|
||||
incomplete walk must not misreport the denominator).
|
||||
same POSIX *rel* the document rows use, plus the position pair the UI's
|
||||
running count consumes. The pair's MEANING is the phase-194 switch
|
||||
``llm.settings.sync_pending_count`` (read ONCE before the source loop —
|
||||
see the pre-scan paragraph below):
|
||||
|
||||
* switch truthy (the default): ``total`` = the size of the pre-scan's
|
||||
PENDING set (the files predicted to be re-indexed) and ``done`` = the
|
||||
1-based position in THAT set. The position advances for every file in
|
||||
the pending set — whether or not the file turns out changed at
|
||||
processing time — so count and denominator stay on the SAME set and
|
||||
the arithmetic never runs away; a file NOT in the set does not advance
|
||||
it, which is what keeps the label holding at ``K remaining`` while the
|
||||
walk moves over the unchanged ``N − K``.
|
||||
* switch ``0``: the exact phase-193 walk position — ``done`` = the
|
||||
1-based index of the current file **across all sources** and ``total``
|
||||
= the combined pre-walk count of importable files (the full rollback).
|
||||
|
||||
In both states the count-only pass happens **only when *progress* is
|
||||
provided**: callers passing nothing pay no extra pass and behave exactly
|
||||
as before (the CLI, the upload job). Under ``limit``, the hook still
|
||||
fires per processed file only — ``done`` never exceeds the limit, but
|
||||
``total`` stays the full count (an incomplete walk must not misreport
|
||||
the denominator).
|
||||
|
||||
The pair is a PREDICTION, and its drift edges are bounded (phase 194
|
||||
A1): a file the scan predicted and that reverted to unchanged before the
|
||||
loop reached it still advances the count (it was counted in ``total``,
|
||||
so the count still reaches 0); a file changed AFTER the scan is indexed
|
||||
for real but advances nothing (``total`` never counted it), and the
|
||||
UI's ``total - done > 0`` guard hides that excess — never a negative and
|
||||
never a stale ``0 remaining``. The terminal ``counts_line`` ("N added ·
|
||||
M updated …") stays the exact surface, untouched.
|
||||
|
||||
``on_scan_progress`` (phase 194, task 01) is an optional SCAN-position
|
||||
hook for the pre-scan itself: while the pending set is being predicted it
|
||||
is called — once per importable file, after that file's hash+read — with
|
||||
``(position_1based, scan_total)``, the file's 1-based position in the
|
||||
scan and the combined importable count across all *sources* roots (the
|
||||
number the phase-64 stats pre-walk used to compute). It carries counts
|
||||
only, never the file identity (the sibling convention), and it runs on
|
||||
the span's WORKER thread — the house rule that these hooks only assign
|
||||
fields is what makes each call a single GIL-atomic store. Same "only
|
||||
when provided" rule as ``progress``, ``on_chunk_progress`` and
|
||||
``on_image_progress``, with one addition: the pre-scan it reports on is
|
||||
gated on *progress* too, so a caller passing ``on_scan_progress`` alone
|
||||
(``progress=None``) runs no scan, fires no callback and stays
|
||||
byte-identical — the scan exists only to serve the running counts.
|
||||
|
||||
``on_chunk_progress`` (phase 129, task 02) is an optional
|
||||
chunk-position hook for the file currently being indexed: while that
|
||||
@@ -923,6 +1094,45 @@ async def import_sources(
|
||||
(``0`` = never — the exact pre-phase behaviour, the operator's
|
||||
no-redeploy rollback). The cadence changes no count, no log line and no
|
||||
committed byte: see the safety argument at the reset site.
|
||||
|
||||
The pre-scan (phase 194, task 01): with the hook on and
|
||||
``llm.settings.sync_pending_count`` truthy, the count-only pass is no
|
||||
longer a stats walk but :func:`_prescan_pending` — every importable file
|
||||
hashed once and compared once against its stored ``file_hash``, giving
|
||||
the PENDING set the ``progress`` pair counts. Three costs the phase
|
||||
accepted, deliberately:
|
||||
|
||||
* **off the loop** (the phase-179 rule): the scan runs in ONE
|
||||
:func:`asyncio.to_thread` span. Ten thousand streamed hashes plus ten
|
||||
thousand one-column SELECTs inline would park the single uvicorn loop
|
||||
for seconds — the exact defect class
|
||||
``tests/integration/test_sync_git_freeze.py`` pins for the git clone,
|
||||
and the ``GET /api/sync/status`` poll the UI runs every 2 s is the
|
||||
victim. A cancel registered at the span's await interrupts the RUN
|
||||
(phase 134: it lands the ``cancelled`` terminal), while the worker
|
||||
itself runs to completion — threads are not interruptible, which is
|
||||
also why the span reads through a session of its own instead of the
|
||||
run's (see :func:`_prescan_pending`).
|
||||
* **one extra read + hash + scalar SELECT per importable file per run**
|
||||
(A3): the second read of the same file is served from the page cache,
|
||||
and the scalar read is a unique ``(source, path)`` index probe.
|
||||
``BOR_SYNC_PENDING_COUNT=0`` buys the whole cost back and restores the
|
||||
phase-193 walk-position semantics byte-for-byte; a negative value
|
||||
refuses startup naming the field.
|
||||
* **the file is hashed twice**, here and again in ``_index_file``. That
|
||||
is deliberate: the pre-scan's digest is the PREDICTION, the loop's is
|
||||
the TRUTH — a file edited between the two is still detected and
|
||||
re-indexed. Dropping the second hash would make the count a claim
|
||||
about bytes nobody verified.
|
||||
|
||||
Everything else in the run is untouched by the scan: ``summary.files``
|
||||
(the ``import: summary … files=`` count of every file the loop
|
||||
processed), ``seen``, the prune post-pass, the date refresh and every log
|
||||
line keep their exact shape — the pre-scan builds ``pending``, never
|
||||
``seen``, and the loop keeps building ``seen`` from disk exactly as
|
||||
before. The degenerate empty knowledge base is the phase-64 shape: scan
|
||||
total ``0``, no callback fires, the pending set is empty and ``total``
|
||||
is ``0`` (the UI's ``total === 0`` guard then renders no count part).
|
||||
"""
|
||||
if limit is not None and limit <= 0:
|
||||
raise ValueError("limit must be >= 1")
|
||||
@@ -937,30 +1147,68 @@ async def import_sources(
|
||||
# reads ``llm.settings`` per RUN, never per file). ``0`` disables the
|
||||
# release entirely.
|
||||
reset_every = llm.settings.sync_session_reset_files
|
||||
# Phase 194 (task 01): the pending-count switch, read ONCE here — before
|
||||
# the source loop — so a settings overlay saved mid-run cannot change the
|
||||
# MEANING of the hook's ``done``/``total`` pair between two files of the
|
||||
# same walk (the phase-184 read-once discipline). ``0`` = the exact
|
||||
# phase-193 walk-position semantics, the operator's no-redeploy rollback.
|
||||
count_pending = bool(llm.settings.sync_pending_count)
|
||||
owns_session = session is None
|
||||
if session is None:
|
||||
session = SessionLocal()
|
||||
seen: set[tuple[str, str]] = set()
|
||||
source_names: set[str] = set()
|
||||
# phase 64 (task 01): the hook's combined denominator, walked with the
|
||||
# exact same rules as the processing loop below (directory stats only,
|
||||
# no file reads). Skipped entirely for ``progress=None`` callers — no
|
||||
# extra pass, byte-identical behaviour and cost.
|
||||
# phase 64 (task 01): the hook's denominator. phase 194 (task 01): what
|
||||
# the number MEANS follows ``count_pending`` — the size of the predicted
|
||||
# change set, or the count of every importable file. Either way the pass
|
||||
# that computes it is skipped entirely for ``progress=None`` callers (no
|
||||
# extra cost, byte-identical behaviour) and it is INSIDE the try so the
|
||||
# session is closed on every exit path.
|
||||
total = 0
|
||||
if progress is not None:
|
||||
for root in sources:
|
||||
total += len(
|
||||
iter_importable_files(
|
||||
root,
|
||||
llm.settings.import_extension_set,
|
||||
ignore=_ignore_for_root(root, ignore_by_root),
|
||||
include_hidden=_include_hidden_for_root(
|
||||
root, include_hidden_by_root
|
||||
),
|
||||
image_extensions=image_exts,
|
||||
)
|
||||
)
|
||||
# (source, rel) -> 1-based position in the PENDING set; ``None`` = the
|
||||
# phase-193 walk-position semantics (the switch ``0`` branch below).
|
||||
pending: dict[tuple[str, str], int] | None = None
|
||||
# the position the hook last reported: a file the scan did not predict
|
||||
# holds it, which keeps ``total - done`` honest while the walk moves over
|
||||
# the unchanged files.
|
||||
pending_done = 0
|
||||
try:
|
||||
if progress is not None:
|
||||
if count_pending:
|
||||
# THE PRE-SCAN (phase 194, D1) — it REPLACES the stats-only
|
||||
# pre-walk, so there is no third walk. ONE off-loop span
|
||||
# (the phase-179 rule: a 10,000-file hash+SELECT pass inline
|
||||
# would park the single uvicorn loop and freeze the status
|
||||
# poll the UI runs every 2 s). The scan total is the scan
|
||||
# hook's own business; the run's denominator is the pending
|
||||
# set the scan returns.
|
||||
_scan_total, pending = await asyncio.to_thread(
|
||||
_prescan_pending,
|
||||
sources,
|
||||
llm.settings.import_extension_set,
|
||||
image_exts,
|
||||
ignore_by_root,
|
||||
include_hidden_by_root,
|
||||
on_scan_progress,
|
||||
)
|
||||
total = len(pending)
|
||||
else:
|
||||
# ``BOR_SYNC_PENDING_COUNT=0`` — the phase-193 rollback,
|
||||
# byte-for-byte: the hook's combined denominator, walked with
|
||||
# the exact same rules as the processing loop below (directory
|
||||
# stats only, no file reads).
|
||||
for root in sources:
|
||||
total += len(
|
||||
iter_importable_files(
|
||||
root,
|
||||
llm.settings.import_extension_set,
|
||||
ignore=_ignore_for_root(root, ignore_by_root),
|
||||
include_hidden=_include_hidden_for_root(
|
||||
root, include_hidden_by_root
|
||||
),
|
||||
image_extensions=image_exts,
|
||||
)
|
||||
)
|
||||
for root in sources:
|
||||
if not root.is_dir():
|
||||
logger.warning("import: source dir not found, skipping: %s", root)
|
||||
@@ -1036,7 +1284,16 @@ async def import_sources(
|
||||
# file that then errors or turns out unchanged was
|
||||
# already the "current file". No try/except around the
|
||||
# call: the hooks in this repo only assign fields.
|
||||
progress(source, rel, summary.files, total)
|
||||
# phase 194 (task 01): the pair's meaning follows the
|
||||
# switch read before the loop — the position in the
|
||||
# pending set, or the pre-phase-194 walk position.
|
||||
if pending is not None:
|
||||
predicted = pending.get((source, rel))
|
||||
if predicted is not None:
|
||||
pending_done = predicted
|
||||
progress(source, rel, pending_done, total)
|
||||
else:
|
||||
progress(source, rel, summary.files, total)
|
||||
try:
|
||||
await _index_file(
|
||||
session, source=source, rel=rel, full_path=path, llm=llm,
|
||||
|
||||
Reference in New Issue
Block a user