bf64c0d7e4077801486717063d5ac4b893927b13
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ducoterra
|
ffa919b8bf |
fix(chat): keep in-flight answers alive across in-app view switches
Root cause (owner repro, verified in a real browser 2026-09-06): the five navbar views (Chat, RAG, Sources, Tuning, History) were separate HTML documents, so a navbar click was a REAL cross-document navigation — the chat page unloaded, the in-flight SSE fetch was aborted, and the phase-48 teardown (app/api/chat.py `finally`, "chat: turn cancelled") stopped the model. Observed: send question -> click RAG mid-stream -> click Chat -> the answer never finished: no `query_log` row, and on return a dangling question with no brain record (the pre-token pagehide partial persist skips because `acc` is empty). Phase-48 LOCKED-DECISION REFINEMENT (owner-confirmed 2026-09-06, flagged per AGENTS.md rule 3, not silently deviated): "real navigation cancels the fetch" now means LEAVING THE APP — tab close, external/other-document navigation, the Stop button. In-app navbar switches are client-side view switches and no longer cancel. Fix — Option A (SPA shell), chosen over B (Service Worker owns the stream) and C (server-side turn registry + resume): - frontend/index.html is the shell: ONE `<main id="main">` holds the five `<section class="view">` blocks; hidden views carry BOTH `hidden` and `inert` (WCAG — no focus/keyboard traversal). The shared header, the single `doc-modal-*` skeleton, and the `#app-version` footer each exist exactly once; the per-view copies from the four folded pages are dropped. - New frontend/assets/router.js (vanilla module — no framework, no bundler, No-CDN rule intact): lazy-imports a view module on FIRST show only (mount-once, hide-forever — the chat view's in-flight SSE reader persists across switches; that persistence IS the fix); intercepts same-shell navbar links with preventDefault + history.pushState (never a document load); handles popstate; single writer of `.nav-link` active state (is-active + aria-current), document.title, and the per-view meta description (values carried over from the old pages' heads, brand-resolved at write time). - Each folded page's JS becomes `export async function mount(root)` — root-scoped queries; `initSharedHeader()` dropped (the header boots once in the shell via the chat module; the admin flag comes from the same cached `fetchIsAdmin()` promise — zero extra requests). - app/main.py: a small list-driven route factory serves the shell for /tuning.html, /sources.html, /git-sources.html, /history.html — registered AFTER the API routers and BEFORE the static catch-all (routes-first). The phase-33 caching middleware applies no-cache + `?v=` rewriting unchanged; app/core/caching.py needed NO change (the view paths did not change — pinned by the integration tests). - The four old view .html files are DELETED (one source of truth); deep links to the old URLs keep working (the router picks the view from the pathname); `/?chat=<id>` is unaffected; the Containerfile bundles router.js (inlining the lazy view modules) and drops the folded page files. - app/schemas.py: HistoryTurn.text cap 4000 -> 32000 — the shell keeps long saved answers in the chat, and the old cap (stricter than the 24_000-char total history budget) 422-rejected any second turn in such a chat (found by the phase-42 E2E suite on the shell). Boundaries: login.html, shared.html, doc-edit.html, document.html REMAIN separate documents (flow pages, not navbar tabs); a mid-stream navigation to doc-edit/document.html still cancels per phase 48 (follow-up candidate, out of scope). The SSE API is unchanged. Real departures still cancel the turn — phase 48 intact (pinned by tests/e2e/test_stop_generation.py, unchanged, and by the new suite's real-departure control). Tests: - Phase-20 suite REWRITTEN to the new semantics (tests/e2e/test_sources_midstream_bug.py): a navbar switch no longer cancels — the stream survives the switch and the FULL answer settles; the pagehide partial persist REMAINS for real departures (the partial's exact shape — first streamed chunk prefix, no done metadata — is still pinned there). - NEW story suite tests/e2e/test_nav_switch_keeps_stream.py (mock LLM): the owner repro (send -> RAG mid-stream -> Chat: window sentinel survives = same document, FULL answer, exactly one brain turn in bor.chat.v1, exactly one settled query_log row, auto-saved row matches) + the same mid-stream switch against the other three views + the real-departure-still-cancels control + the no-switch baseline. - tests/unit/test_frontend_router.py: source-level pins of the router invariants (click interceptor targets ONLY same-shell view paths, pushState-only switches, mount-once guard, hidden+inert pair, single-writer active state/title); shell-route integration tests (each folded path serves the shell with no-cache + `?v=` body; a non-view path still 404s); the file-reading unit pins re-pointed at the shell (the four view files are gone — the shell is the source of truth). Verification (this commit): full suite green — 1565 unit+integration tests, app/ coverage 99% (>90% floor); ruff + pyright clean; the phase's E2E suites green in isolation (house protocol, AGENTS.md rule 9). Owner repro verified in a real browser against the real LLM (dev server :8010, headful Chromium): "tell me about everquest" -> RAG mid-stream -> Chat — the answer completed with one brain bubble and no error banner, `query_log` gained exactly one settled row (deflected=True: the dev KB holds no EverQuest docs — the settle, not the topic, is the proof), zero "chat: turn cancelled" lines for that turn; the control (real navigation to /shared.html mid-stream) still cancelled (no settled row, the cancel line logged, the partial persisted on return). Screenshots: .agents/screenshots/76_manual_*. Phase 76 (76_spa_nav_shell) complete — moved to .agents/phases/complete/. |
||
|
ducoterra
|
dbf2af26c6 |
refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills): phases/, user_stories/, reports/, screenshots/, validate.sh, and phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves history; runtime artifacts move alongside). Updates every reference in AGENTS.md, README.md, .gitignore, app docstrings, and test story headers. Historical KB content in data/ and the runtime pipeline.log transcript are left untouched. |
||
|
ducoterra
|
03d26255c6 |
feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan … reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content" — design confirmed in the same conversation): * POST /api/git-sources/upload (admin-only, require_admin): accepts .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap (bounds BOTH the compressed upload and the total extracted bytes — zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink escape and device/FIFO members rejected), and atomically swaps the content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the archive suffix — no missing window, a failed upload never touches the existing folder/row/KB). The git_sources row is upserted by path (kind='local', no duplicates, added_at preserved), the models are checked fail-fast (503 sanitized when down — the folder/row stay committed and the next sync/re-upload retries idempotently), and the source is scanned synchronously in the request (single-source import_sources prune=True + change-gated KB overview), answering 200 with the sync-style counts. One upload at a time (409); the request session is released before the scan so a concurrent TRUNCATE cannot deadlock against it. * app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES, archive_source_name (safe-name derivation), unpack_archive (guarded zip/tar extraction with the extracted-byte cap, no partial state), swap_in (atomic replace with restore-on-failure) — fully unit-tested. * app/config.py + .env.example: BOR_UPLOAD_DIR (default ~/bor-sources/uploads, deliberately separate from the git checkouts) and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at startup on <= 0). * python-multipart added to the dependencies — FastAPI's required multipart parser (an A2 implementation detail, phase locked decision). * The Sources page: the phase-38 "Add a local directory" form is removed; #archive-upload-form takes its place (labeled file input, "Upload & scan" button, the §7.4 never-stale lifecycle, inline role=alert error, role=status count line); hint + table caption updated. The POST /api/git-sources kind=local API contract is UNCHANGED — a plain directory is still registrable via the API, and existing Local rows list/remove/sync exactly as before. * The phase-38 story E2E (test_local_directory_sources.py) is rewritten API-driven — the form it drove is gone; its acceptance stands. * The story E2E (test_archive_upload_sources.py): the swap, upload→scan→list (the deterministic "Uploading…" in-flight state, the Local row, /api/docs + the RAG catalog), same-filename re-upload (in-place replace, prune, no duplicate row, v2-only folder), the 422 inline error + recovery (the form is not wedged), and the anonymous gate + 403. * README: the archive-upload section (formats, naming rule, in-place replace, both new settings), the local-directory form removal noted, config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB. Gates: unit+integration green, app/ coverage 99%, the story E2E green in isolation, the regression suites (git sources admin, local directory sources, sync button, import documents, nav rename, smoke, shared header) green in isolation, ruff + pyright clean. Note: per this phase's file-level staging, frontend/assets/styles.css also carries the small same-day in-flight owner rework already in the working tree (the .sign-in-mobile companion rule for the phase-48 mobile sign-in copy); the phase-49 change is the upload form's block. |
||
|
ducoterra
|
94d7228510 |
feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
|