phase: 194_sync_pending_count
**Phase 194 — final verification pass: all green.**
**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.
**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.
**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.
**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.
**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
This commit is contained in:
+163
-21
@@ -103,7 +103,16 @@ carries the phase-64 per-file progress — ``current_file`` (the
|
||||
``source/relative/path`` the import is processing right now) plus
|
||||
``files_done`` / ``files_total`` — null/0/0 before the import starts
|
||||
(clone/pull reports no file yet) and in terminal states, which clear
|
||||
``current_file`` but keep the run's final counts.
|
||||
``current_file`` but keep the run's final counts. What the pair COUNTS
|
||||
changed in phase 194: on a progress caller (the UI sync is the only
|
||||
one) it is the position in the pre-scan's PREDICTED change set —
|
||||
``files_total`` the number of files that will actually be re-indexed,
|
||||
``files_done`` the 1-based position in that set — which is what makes
|
||||
the Sources page's ``N remaining`` say "10 remaining" on a ~10,000-file
|
||||
KB with 10 changes (the owner report of 2026-10-19) instead of counting
|
||||
the walk. ``BOR_SYNC_PENDING_COUNT=0`` restores the phase-193 walk
|
||||
position byte-for-byte; the terminal ``detail`` counts stay the exact
|
||||
surface either way.
|
||||
|
||||
Phase 134 (task 01) makes the machine FIVE-state: ``cancelled`` joins
|
||||
``success`` / ``failed`` as a TERMINAL state — it carries
|
||||
@@ -131,7 +140,14 @@ checks the event at the pipeline boundaries via
|
||||
set): the top of the per-source loop (before each clone/pull or
|
||||
local-dir check), immediately before ``import_sources``, and
|
||||
immediately before the overview + summaries span (the
|
||||
``added + updated > 0`` block). Consequence a future reader must
|
||||
``added + updated > 0`` block). Phase 194 (task 02) adds one more
|
||||
await INSIDE the import span for the same shape: the pre-scan runs in
|
||||
a ``to_thread`` worker, so a cancel parked there unblocks the RUN at
|
||||
once (the ``cancelled`` terminal, nothing indexed) while the scan
|
||||
thread finishes the session of its own — the phase-134/179 split, the
|
||||
worker not being interruptible, pinned by
|
||||
``tests/integration/test_sync_pending_count.py`` (importer side) and
|
||||
this module's cancel-during-scan pin. Consequence a future reader must
|
||||
know: a git subprocess already running at the cancel moment runs to
|
||||
completion (Python threads are not interruptible — since phase 179
|
||||
it occupies a ``to_thread`` worker rather than the loop, so it no
|
||||
@@ -188,8 +204,17 @@ background task must die in state, never as an unobserved exception
|
||||
|
||||
Phase 98 (task 01) adds the run's PHASE machine — ``phase`` is the
|
||||
pipeline stage the run is in: the model-check + clone/pull prelude
|
||||
reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
|
||||
(set immediately before ``import_sources``), ``"overview"`` (set
|
||||
reports ``null`` (the bare "Syncing…" label stays), then ``"scan"``
|
||||
(phase 194, task 02 — set immediately before ``import_sources``,
|
||||
REPLACING the pre-phase ``"import"`` set: what happens first in that
|
||||
call is the pre-scan that predicts the change set, and the label the
|
||||
task-03 branch renders for this phase is ``Scanning… (n/total)``), then
|
||||
``"import"`` (flipped by
|
||||
the FIRST per-file ``progress`` call while the phase is still
|
||||
``"scan"`` — the flip lives in the runner's own ``_hook`` closure, so
|
||||
the importer gains no extra callback; with
|
||||
``BOR_SYNC_PENDING_COUNT=0`` no scan report fires at all and the flip
|
||||
is simply the first file), ``"overview"`` (set
|
||||
before ``regenerate_overview`` — a changed KB only), and
|
||||
``"summaries"`` (set before a folder-summary generation — the
|
||||
changed-KB incremental regeneration (phase 127, task 03 — the changed
|
||||
@@ -259,7 +284,12 @@ sampled through the ONE :func:`_sample_rss` — at the run start, at every
|
||||
``phase`` transition, in every terminal, and at most once per
|
||||
``_RSS_SAMPLE_INTERVAL_S`` from the per-file progress hook, so a 27k-file
|
||||
walk costs ~600 ``/proc`` reads and not 27k — LOCKED A4), and
|
||||
``GET /api/sync/status`` reports both keys ADDITIVELY. The terminals keep
|
||||
``GET /api/sync/status`` reports both keys ADDITIVELY. Phase 194
|
||||
(task 02) extends both halves of that rule to the new stage: the
|
||||
``"scan"`` set before ``import_sources`` and the ``scan``→``import``
|
||||
flip are phase TRANSITIONS and FORCE the read, while the pre-scan's own
|
||||
per-file hook samples UNFORCED (throttled) — a 10,000-file scan costs
|
||||
one ``/proc`` read per interval, never one per file. The terminals keep
|
||||
the FINAL reading next to the final counts (the phase-64 convention), so
|
||||
the page that started the run answers the incident's question without a
|
||||
shell. A reading the platform cannot know stays ``null`` on the wire (the
|
||||
@@ -323,11 +353,33 @@ class SyncStatus:
|
||||
outside the import phase — clone/pull first, terminal states
|
||||
after); ``files_done`` / ``files_total`` carry the hook's
|
||||
done/total position and survive a terminal state (the run's last
|
||||
position is useful context next to the error).
|
||||
position is useful context next to the error). Since phase 194 the
|
||||
pair is the hook's PENDING position (phase 194: the pre-scan's
|
||||
predicted change set — ``files_total`` the files that will actually
|
||||
be re-indexed), not the walk position; ``BOR_SYNC_PENDING_COUNT=0``
|
||||
restores the walk position and every other caller (the CLI, the
|
||||
upload job) is unaffected.
|
||||
|
||||
Phase 194 (task 02) scan fields: ``scan_done`` / ``scan_total``
|
||||
carry the pre-scan's position — the 1-based file whose hash+read
|
||||
just completed over the combined importable count the scan walks
|
||||
(the importer's ``on_scan_progress`` hook). They are 0/0 before the
|
||||
scan starts and on the ``BOR_SYNC_PENDING_COUNT=0`` rollback (no
|
||||
scan, so no report), reset with the run like every other field, and
|
||||
KEPT at every terminal beside the final file counts (the phase-64
|
||||
keep-final-values convention — the scan the owner watched is still
|
||||
on the page after the run). ``current_file`` stays ``null`` through
|
||||
the phase: the scan names no file, it only counts. The degenerate
|
||||
empty knowledge base is the honest edge — the scan predicts nothing
|
||||
so no report fires and no ``progress`` call ever arrives: the phase
|
||||
stays ``"scan"`` until the next transition or the terminal (which
|
||||
clears it), and the label is the bare phase name with no count part
|
||||
(the UI's ``total === 0`` guard).
|
||||
|
||||
Phase 98 (task 01) phase fields: ``phase`` is the pipeline stage
|
||||
(null in the model-check + clone/pull prelude and in terminal
|
||||
states — the module docstring's phase machine); while in the
|
||||
states — the module docstring's phase machine, whose stages are now
|
||||
``scan`` → ``import`` → ``overview`` → ``summaries``); while in the
|
||||
``"summaries"`` phase, ``current_summary`` is the source /
|
||||
``source/folder_path`` the generator is summarizing right now and
|
||||
``summaries_done`` / ``summaries_total`` carry the generator's
|
||||
@@ -393,10 +445,20 @@ class SyncStatus:
|
||||
current_file: str | None = None
|
||||
files_done: int = 0
|
||||
files_total: int = 0
|
||||
# Phase 194 (task 02): the PRE-SCAN's position — the 1-based file
|
||||
# whose hash+read just completed and the combined importable count
|
||||
# the scan walks (the importer's on_scan_progress hook). 0/0 before
|
||||
# the scan starts (and on the BOR_SYNC_PENDING_COUNT=0 rollback);
|
||||
# reset with the run and kept at every terminal beside the final
|
||||
# file counts (the phase-64 keep-final-values convention).
|
||||
scan_done: int = 0
|
||||
scan_total: int = 0
|
||||
# Phase 98 (task 01): the phase machine — the pipeline stage the
|
||||
# run is in and, while in the folder-summary phase, the folder
|
||||
# being summarized plus the hook's done/total position.
|
||||
phase: Literal["import", "overview", "summaries"] | None = None
|
||||
# being summarized plus the hook's done/total position. Phase 194
|
||||
# (task 02) adds the "scan" stage (the pre-scan that predicts the
|
||||
# run's change set — the module docstring's phase machine).
|
||||
phase: Literal["scan", "import", "overview", "summaries"] | None = None
|
||||
current_summary: str | None = None
|
||||
summaries_done: int = 0
|
||||
summaries_total: int = 0
|
||||
@@ -433,13 +495,15 @@ _task: asyncio.Task[None] | None = None
|
||||
# starts, so a stale event can never bleed into the next run.
|
||||
_cancel_event = asyncio.Event()
|
||||
|
||||
#: Phase 187 (task 03, D3/A4): how often the PER-FILE progress hook may
|
||||
#: re-read the process memory, in seconds. That hook fires once per
|
||||
#: file, so without this guard a 27k-file walk would cost 27k
|
||||
#: ``/proc/self/status`` reads; with it the same walk costs ~600 and the
|
||||
#: operator watching the page still sees the number move (PLAN §7.4's
|
||||
#: never-stale contract). Every other sampling point (the run start, the
|
||||
#: ``phase`` transitions, the terminals) FORCES the read instead.
|
||||
#: Phase 187 (task 03, D3/A4): how often the PER-FILE progress hooks may
|
||||
#: re-read the process memory, in seconds. Those hooks fire once per
|
||||
#: file — the import's ``progress`` AND, since phase 194 (task 02), the
|
||||
#: pre-scan's ``on_scan_progress`` — so without this guard a 27k-file
|
||||
#: walk would cost 27k ``/proc/self/status`` reads; with it the same walk
|
||||
#: costs ~600 and the operator watching the page still sees the number
|
||||
#: move (PLAN §7.4's never-stale contract). Every other sampling point
|
||||
#: (the run start, the ``phase`` transitions including the ``scan`` set
|
||||
#: and its flip to ``import``, the terminals) FORCES the read instead.
|
||||
_RSS_SAMPLE_INTERVAL_S = 5.0
|
||||
|
||||
#: ``time.monotonic()`` of the last RSS read; ``-inf`` means "never", so
|
||||
@@ -503,7 +567,21 @@ def sync_status() -> dict[str, Any]:
|
||||
import is processing right now — null during the clone/pull phase
|
||||
and in terminal states; ``files_done`` / ``files_total`` carry the
|
||||
hook's position (0/0 idle) and keep their final values in the
|
||||
terminal. ``current_chunk`` / ``chunks_total`` (phase 129) are the
|
||||
terminal — the position in the pre-scan's PREDICTED change set on a
|
||||
progress caller (phase 194), the walk position under
|
||||
``BOR_SYNC_PENDING_COUNT=0``. ``scan_done`` / ``scan_total`` (phase
|
||||
194) are the PRE-SCAN's position: the 1-based file whose
|
||||
hash+compare just ran over the combined importable count the scan
|
||||
walks. They are 0/0 before the scan starts — the prelude, a run on
|
||||
the ``0`` rollback, and every run that never happened — and in a
|
||||
terminal they keep the FINAL scan values beside the final file
|
||||
counts (the phase-64 keep-final-values convention); ``current_file``
|
||||
stays null while ``phase`` is ``"scan"``, because the scan names no
|
||||
file. Both keys are ADDITIVE — placed after ``files_total``, every
|
||||
pre-existing key keeping its name, type and position. The degenerate
|
||||
empty knowledge base reports the phase without ever reporting a
|
||||
position (0/0 — the UI's ``total === 0`` guard renders no count
|
||||
part). ``current_chunk`` / ``chunks_total`` (phase 129) are the
|
||||
in-flight file's embed-chunk position (batch granularity — the
|
||||
first chunk of the in-flight batch over the file's total content
|
||||
chunks): 0/0 before the first batch, in the prelude/overview/
|
||||
@@ -551,6 +629,14 @@ def sync_status() -> dict[str, Any]:
|
||||
"current_file": _status.current_file,
|
||||
"files_done": _status.files_done,
|
||||
"files_total": _status.files_total,
|
||||
# Phase 194 (task 02): the pre-scan's position (the importer's
|
||||
# on_scan_progress hook) — 0/0 before the scan starts and on the
|
||||
# BOR_SYNC_PENDING_COUNT=0 rollback, the FINAL pair kept in every
|
||||
# terminal beside the file counts above (the keep-final-values
|
||||
# convention). Additive: every pre-existing key keeps its name,
|
||||
# type and position.
|
||||
"scan_done": _status.scan_done,
|
||||
"scan_total": _status.scan_total,
|
||||
# Phase 129 (task 03): the in-flight file's embed-chunk
|
||||
# position — the dataclass defaults make idle 0/0, and the
|
||||
# runner clears it at every point the import phase ends
|
||||
@@ -672,6 +758,14 @@ async def _run_sync() -> None:
|
||||
so a long clone no longer stalls the rest of the HTTP surface (the
|
||||
module docstring's freeze paragraph); the run's own sequencing is
|
||||
byte-for-byte the pipeline above.
|
||||
|
||||
Phase 194 (task 02): the phase set before ``import_sources`` is
|
||||
``"scan"`` (not ``"import"``), because that call begins with the
|
||||
off-loop pre-scan that predicts the change set; the FIRST per-file
|
||||
``progress`` call flips it to ``"import"`` inside ``_hook``. Both
|
||||
transitions force the memory sample, the scan's own per-file hook
|
||||
does not (LOCKED A4), and ``scan_done`` / ``scan_total`` reset with
|
||||
the run and survive every terminal beside the file counts.
|
||||
"""
|
||||
_status.state = "running"
|
||||
_status.started_at = datetime.now(UTC)
|
||||
@@ -683,6 +777,11 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None
|
||||
_status.files_done = 0
|
||||
_status.files_total = 0
|
||||
# Phase 194 (task 02): the pre-scan's position resets with the run —
|
||||
# 0/0 until the scan reports its first file (and 0/0 for the whole
|
||||
# run on the BOR_SYNC_PENDING_COUNT=0 rollback, which scans nothing).
|
||||
_status.scan_done = 0
|
||||
_status.scan_total = 0
|
||||
# Phase 98 (task 01): the phase fields reset with the run — no
|
||||
# phase until the import starts (the model-check + clone/pull
|
||||
# prelude reports null), no folder until the summary span starts.
|
||||
@@ -834,6 +933,17 @@ async def _run_sync() -> None:
|
||||
# run-start reset alone cannot cover it).
|
||||
_status.images_done = 0
|
||||
_status.images_total = 0
|
||||
# Phase 194 (task 02): the scan→import FLIP. The phase is
|
||||
# set to "scan" immediately before import_sources, and the
|
||||
# first file the walk actually reaches is where the running
|
||||
# label stops being a scan — no extra callback reaches the
|
||||
# importer, the flip lives in this existing per-file hook.
|
||||
if _status.phase == "scan":
|
||||
_status.phase = "import"
|
||||
# A phase TRANSITION is a forced sampling point (phase
|
||||
# 187), so this one read is not the throttled per-file
|
||||
# one the tail of this closure does.
|
||||
_sample_rss(force=True)
|
||||
# Phase 187 (task 03): the run's memory, sampled THROTTLED
|
||||
# (unforced) — this hook fires once per file, so a 27k-file
|
||||
# walk reads /proc ~600 times instead of 27k (LOCKED A4)
|
||||
@@ -865,10 +975,30 @@ async def _run_sync() -> None:
|
||||
_status.images_done = done
|
||||
_status.images_total = total
|
||||
|
||||
# Phase 98 (task 01): the phase machine — the import phase
|
||||
# starts NOW (the model-check + clone/pull prelude above
|
||||
# reported ``phase: null``).
|
||||
_status.phase = "import"
|
||||
# Phase 194 (task 02): the PRE-SCAN's position — the importer's
|
||||
# ``on_scan_progress`` hook, fired once per importable file while
|
||||
# the run's change set is still being PREDICTED (before the walk),
|
||||
# with (position_1based, scan_total). It runs on the scan's
|
||||
# WORKER thread, so this closure only assigns fields on the
|
||||
# module's own status object — the house hook contract that makes
|
||||
# each call a single GIL-atomic store (phase 179). The memory
|
||||
# sample is UNFORCED here: this fires once per file, exactly like
|
||||
# ``_hook`` above, so it costs one /proc read per interval and
|
||||
# never one per file (LOCKED A4).
|
||||
def _scan_hook(scan: int, total: int) -> None:
|
||||
_status.scan_done = scan
|
||||
_status.scan_total = total
|
||||
_sample_rss()
|
||||
|
||||
# Phase 98 (task 01) + phase 194 (task 02): the phase machine —
|
||||
# the SCAN phase starts NOW (the model-check + clone/pull prelude
|
||||
# above reported ``phase: null``). What runs first inside
|
||||
# ``import_sources`` is the pre-scan that predicts the change set,
|
||||
# and the label the owner reads through it is "Scanning…"
|
||||
# (phase 193's "import" set moved to the first ``_hook`` call
|
||||
# above — the flip), so the count the page shows is never a scan
|
||||
# wearing the work's shape.
|
||||
_status.phase = "scan"
|
||||
# Phase 187 (task 03): the phase transitions are the forced
|
||||
# sampling points — each stage of the run begins with its own
|
||||
# reading on the status (the per-file hook throttles).
|
||||
@@ -885,6 +1015,7 @@ async def _run_sync() -> None:
|
||||
progress=_hook,
|
||||
on_chunk_progress=_chunk_hook,
|
||||
on_image_progress=_image_hook, # phase 152 (task 03)
|
||||
on_scan_progress=_scan_hook, # phase 194 (task 02)
|
||||
ignore_by_root=ignore_by_root,
|
||||
include_hidden_by_root=include_hidden_by_root,
|
||||
doc_dates_by_root=doc_dates_by_root,
|
||||
@@ -1034,6 +1165,10 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # phase 64: keep the final counts
|
||||
_status.phase = None # phase 98: keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 194 (task 02): the scan position is NOT cleared here —
|
||||
# like the file counts it keeps its final values beside them
|
||||
# (the keep-final-values convention), so the page reloaded after
|
||||
# the run still says how big the predicted change set was.
|
||||
# Phase 129 (task 03): the chunk position is a sub-position of
|
||||
# a file that's done — it clears in the terminal (unlike the
|
||||
# FILE counts, which keep their final values for the result
|
||||
@@ -1091,6 +1226,10 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # keep the final counts
|
||||
_status.phase = None # keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 194 (task 02): the scan position survives the cancel
|
||||
# too — where a cancel landed DURING the pre-scan it is the
|
||||
# run's only number, and the log line above keeps the file
|
||||
# pair beside it (the keep-final-values convention).
|
||||
# Phase 129 (task 03): the chunk position is a sub-position
|
||||
# of a file that's done — it clears in EVERY terminal, like
|
||||
# success and failed (the convention split, pinned).
|
||||
@@ -1138,6 +1277,9 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # phase 64: keep the final counts
|
||||
_status.phase = None # phase 98: keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 194 (task 02): the scan position is kept here as well —
|
||||
# a run that died mid-scan keeps the position it reached beside
|
||||
# the final file counts (the convention, every terminal).
|
||||
# Phase 129 (task 03): the chunk position clears in the
|
||||
# failed terminal too (the convention split — the FILE counts
|
||||
# keep their final values next to the error, the sub-position
|
||||
|
||||
Reference in New Issue
Block a user