Single consolidated commit for four completed, validated phases (77, 78,
79, 80). The pipeline run left all work uncommitted because the harness
commits only with PHASE_COMMIT=1 while child executors are forbidden from
committing; the phases themselves all passed validation and moved to
.agents/phases/complete/.
Phase 77 — navbar view refresh
- router.js dispatches bor:view-refresh on re-show / active re-click /
popstate (gated on wasMounted; first show and boot exempt)
- History / RAG / Sources / Tuning re-fetch on refresh (admin branch);
Chat deliberately excluded (stream survival)
- History "Refresh" button (admin-only, in-flight disable + status line)
- New story suite tests/e2e/test_navbar_refresh.py (7 tests)
Phase 78 — static background
- Removed the animated glow layers; static 44px grid over the flat --bg
canvas; default and reduced-motion renders byte-identical
- Updated background/theme E2E suites; removed bg-glow test pins
Phase 79 — API tokens
- api_tokens model + migration 0012; hash-only token service
- Admin tokens API + Tokens admin view; POST /api/token-auth;
live-revoking require_user on chat / suggestions / document content
- Frontend token gate with localStorage cache; anonymous E2E suites
migrated to token login
- New story suite tests/e2e/test_api_tokens.py (9 tests)
Phase 80 — history suggestion chips
- last_questions() endpoint with SEED fallback; startNewChat() refetch
- Seed-semantics docs (config.py, .env.example, README)
- Integration state matrix + E2E suite rewritten to the 4 chip states
Also included: phase-76 report artifacts and the repo restore-test-db
skill (previously untracked), scripts/* ruff fixes from phase 77.
Final gate state (phase 80 final pass, covers everything above):
- uv run pytest --cov=app → 1637 passed, 0 failed, app/ coverage 99%
- uv run ruff check . && uv run pyright → clean, 0 errors
- Per-phase story E2E suites green in isolation
Root cause (owner repro, verified in a real browser 2026-09-06): the
five navbar views (Chat, RAG, Sources, Tuning, History) were separate
HTML documents, so a navbar click was a REAL cross-document navigation
— the chat page unloaded, the in-flight SSE fetch was aborted, and the
phase-48 teardown (app/api/chat.py `finally`, "chat: turn cancelled")
stopped the model. Observed: send question -> click RAG mid-stream ->
click Chat -> the answer never finished: no `query_log` row, and on
return a dangling question with no brain record (the pre-token pagehide
partial persist skips because `acc` is empty).
Phase-48 LOCKED-DECISION REFINEMENT (owner-confirmed 2026-09-06,
flagged per AGENTS.md rule 3, not silently deviated): "real navigation
cancels the fetch" now means LEAVING THE APP — tab close,
external/other-document navigation, the Stop button. In-app navbar
switches are client-side view switches and no longer cancel.
Fix — Option A (SPA shell), chosen over B (Service Worker owns the
stream) and C (server-side turn registry + resume):
- frontend/index.html is the shell: ONE `<main id="main">` holds the
five `<section class="view">` blocks; hidden views carry BOTH
`hidden` and `inert` (WCAG — no focus/keyboard traversal). The
shared header, the single `doc-modal-*` skeleton, and the
`#app-version` footer each exist exactly once; the per-view copies
from the four folded pages are dropped.
- New frontend/assets/router.js (vanilla module — no framework, no
bundler, No-CDN rule intact): lazy-imports a view module on FIRST
show only (mount-once, hide-forever — the chat view's in-flight SSE
reader persists across switches; that persistence IS the fix);
intercepts same-shell navbar links with preventDefault +
history.pushState (never a document load); handles popstate; single
writer of `.nav-link` active state (is-active + aria-current),
document.title, and the per-view meta description (values carried
over from the old pages' heads, brand-resolved at write time).
- Each folded page's JS becomes `export async function mount(root)` —
root-scoped queries; `initSharedHeader()` dropped (the header boots
once in the shell via the chat module; the admin flag comes from the
same cached `fetchIsAdmin()` promise — zero extra requests).
- app/main.py: a small list-driven route factory serves the shell for
/tuning.html, /sources.html, /git-sources.html, /history.html —
registered AFTER the API routers and BEFORE the static catch-all
(routes-first). The phase-33 caching middleware applies no-cache +
`?v=` rewriting unchanged; app/core/caching.py needed NO change
(the view paths did not change — pinned by the integration tests).
- The four old view .html files are DELETED (one source of truth);
deep links to the old URLs keep working (the router picks the view
from the pathname); `/?chat=<id>` is unaffected; the Containerfile
bundles router.js (inlining the lazy view modules) and drops the
folded page files.
- app/schemas.py: HistoryTurn.text cap 4000 -> 32000 — the shell
keeps long saved answers in the chat, and the old cap (stricter than
the 24_000-char total history budget) 422-rejected any second turn
in such a chat (found by the phase-42 E2E suite on the shell).
Boundaries: login.html, shared.html, doc-edit.html, document.html
REMAIN separate documents (flow pages, not navbar tabs); a mid-stream
navigation to doc-edit/document.html still cancels per phase 48
(follow-up candidate, out of scope). The SSE API is unchanged. Real
departures still cancel the turn — phase 48 intact (pinned by
tests/e2e/test_stop_generation.py, unchanged, and by the new suite's
real-departure control).
Tests:
- Phase-20 suite REWRITTEN to the new semantics
(tests/e2e/test_sources_midstream_bug.py): a navbar switch no longer
cancels — the stream survives the switch and the FULL answer
settles; the pagehide partial persist REMAINS for real departures
(the partial's exact shape — first streamed chunk prefix, no done
metadata — is still pinned there).
- NEW story suite tests/e2e/test_nav_switch_keeps_stream.py (mock
LLM): the owner repro (send -> RAG mid-stream -> Chat: window
sentinel survives = same document, FULL answer, exactly one brain
turn in bor.chat.v1, exactly one settled query_log row, auto-saved
row matches) + the same mid-stream switch against the other three
views + the real-departure-still-cancels control + the no-switch
baseline.
- tests/unit/test_frontend_router.py: source-level pins of the router
invariants (click interceptor targets ONLY same-shell view paths,
pushState-only switches, mount-once guard, hidden+inert pair,
single-writer active state/title); shell-route integration tests
(each folded path serves the shell with no-cache + `?v=` body; a
non-view path still 404s); the file-reading unit pins re-pointed at
the shell (the four view files are gone — the shell is the source
of truth).
Verification (this commit): full suite green — 1565 unit+integration
tests, app/ coverage 99% (>90% floor); ruff + pyright clean; the
phase's E2E suites green in isolation (house protocol, AGENTS.md rule
9). Owner repro verified in a real browser against the real LLM
(dev server :8010, headful Chromium): "tell me about everquest" ->
RAG mid-stream -> Chat — the answer completed with one brain bubble
and no error banner, `query_log` gained exactly one settled row
(deflected=True: the dev KB holds no EverQuest docs — the settle, not
the topic, is the proof), zero "chat: turn cancelled" lines for that
turn; the control (real navigation to /shared.html mid-stream) still
cancelled (no settled row, the cancel line logged, the partial
persisted on return). Screenshots: .agents/screenshots/76_manual_*.
Phase 76 (76_spa_nav_shell) complete — moved to
.agents/phases/complete/.
Phase 74 (TODO.md L4): a follow-up question now reaches the model WITH
the conversation so far — every prior user/brain turn and the prior
thinking blocks on brain turns (preserve-thinking) — while
POST /api/chat stays stateless (A10): the client provides the history
in the request body and the server stores nothing new.
Server (task 01):
- ChatRequest.history: optional list[HistoryTurn] (who: user|brain,
text, optional thinking) — absent/empty keeps the request
byte-identical to pre-phase-74 (the two-message [system, user]
request; the kill-switch semantics are pinned in the integration
suite).
- app.rag.prompts.history_to_messages: pure mapper — walks the turns
newest-first against the settings budgets (history_max_turns=40 /
history_max_chars=24000, BOR_HISTORY_MAX_TURNS /
BOR_HISTORY_MAX_CHARS); a capped turn is dropped WHOLE (never cut
mid-answer); the kept window is returned oldest-first; brain turns
carry their thinking as reasoning_content (A4) only when
non-empty.
- Both branches feed it: the deflected path splices it between the
system prompt and the current user message (the phase-71 recovery
still rebuilds from messages[1:]), the grounded agent receives
run_agent(..., history=hist); llm.py's message params widen to
list[dict[str, Any]] (string-only messages stay byte-identical on
the wire — the SDK passes message dicts through verbatim).
- The per-turn log line (PLAN §9) gains history_msgs=N after
kb_chars=N.
- Pins: tests/unit/test_history.py (mapper: mapping, reasoning
gating, both budgets, drop-whole, ordering, empty default),
tests/unit/test_config.py (the two settings + env overrides),
tests/unit/test_agent.py (the history splice + the default),
tests/integration/test_chat_api.py (deflected AND grounded forward
the history incl. reasoning_content, no-history byte-identity, 422
pins, the log field).
Client (task 02):
- runTurn — the single funnel for fresh send / phase-49 retry /
phase-53 stale-regen — sends history = the conversation record
minus the current question, with thinking only on brain records
that streamed one (undefined drops the key from the JSON, the
record's convention); the question is never duplicated into the
history.
Wire proof (task 03):
- The mock's echo my history marker (HISTORY_TRIGGER) answers with
the deterministic history echo — history: N prior messages; last
answer tail: <last 24 chars>; thinking: yes|no — checked BEFORE
the DEFLECT_MODE branch (like TABLE_TRIGGER), so it fires on both
turn branches whatever the gate says; the module docstring records
the user/assistant-only history invariant that keeps every
existing (tool-result-classified) marker flow unaffected.
- tests/e2e/test_llm_history.py (isolated): a grounded follow-up and
a deflected follow-up both receive history: 2 prior messages +
thinking: yes + the byte-exact tail of turn 1's answer (derived
from the persisted bor.chat.v1 record — the same array the client
maps into the body); a cold start receives history: 0 prior
messages / last answer tail: none / thinking: no.
- Regressions green in isolation: chat_rag, chat_history (phase 50),
agent_document_tools, harness_aligned_tools, stop_generation,
retry_answer, response_to_docs.
Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):
* POST /api/git-sources/upload (admin-only, require_admin): accepts
.tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
(bounds BOTH the compressed upload and the total extracted bytes —
zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
escape and device/FIFO members rejected), and atomically swaps the
content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
archive suffix — no missing window, a failed upload never touches the
existing folder/row/KB). The git_sources row is upserted by path
(kind='local', no duplicates, added_at preserved), the models are
checked fail-fast (503 sanitized when down — the folder/row stay
committed and the next sync/re-upload retries idempotently), and the
source is scanned synchronously in the request (single-source
import_sources prune=True + change-gated KB overview), answering 200
with the sync-style counts. One upload at a time (409); the request
session is released before the scan so a concurrent TRUNCATE cannot
deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
archive_source_name (safe-name derivation), unpack_archive (guarded
zip/tar extraction with the extracted-byte cap, no partial state),
swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
~/bor-sources/uploads, deliberately separate from the git checkouts)
and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
removed; #archive-upload-form takes its place (labeled file input,
"Upload & scan" button, the §7.4 never-stale lifecycle, inline
role=alert error, role=status count line); hint + table caption
updated. The POST /api/git-sources kind=local API contract is
UNCHANGED — a plain directory is still registrable via the API, and
existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
upload→scan→list (the deterministic "Uploading…" in-flight state, the
Local row, /api/docs + the RAG catalog), same-filename re-upload
(in-place replace, prune, no duplicate row, v2-only folder), the
422 inline error + recovery (the form is not wedged), and the
anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
replace, both new settings), the local-directory form removal noted,
config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.
Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.
Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
Grounded chat turns now run the agent loop (app/rag/agent.py) instead
of a bare chat_stream: while the per-turn budgets last
(BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each) the model
gets list_documents (the indexed catalog, /api/docs order) and
read_document (full text, never truncated — A7-revised contract); once
both budgets are spent the tools key is dropped from the request and
the model must answer. Rejected calls (unknown tool, unknown/missing
path, document already in context, spent budget) consume no budget.
Budgets 0/0 make exactly one tools=None request — byte-identical to
the pre-phase path (budgets-as-kill-switch). Deflected turns keep the
direct chat_stream (A8 unchanged; the LOW prompt never carries the
<tools> section).
SSE contract gains {"type":"tool","name":...,"argument":
"source/path"|null} frames ahead of the answer deltas (PLAN §4
extension, owner permission 2026-08-26); done.sources, query_log.sources
and the per-turn log line (gains tool_calls=N) report the retrieval
docs + read docs, deduped. The UI shows a "calling tool"
button/label state and one visible .tool-call line per call above the
answer; the lines persist with the chat record and re-render on
reload. chat_stream passes tools through and accumulates streaming
tool_calls deltas into ToolCallPiece (tools=None stays byte-identical).
E2E: deterministic mock tool flow ("use your tools" + <tools> marker:
list -> read first catalog line -> quoted answer) plus the story suite
(marker flow, reload re-render, plain/deflected no-tool regressions).
Docs: .env.example + README (the two tools, the budgets, the SSE tool
frame, the "calling tool" UI state).
probe: turbo tool_calls=supported 2026-08-26 (uv run python -m
scripts.llm_probe --tools — non-streaming + streaming
finish_reason=tool_calls, indexed delta.tool_calls partials)
ducoterra