Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):
* POST /api/git-sources/upload (admin-only, require_admin): accepts
.tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
(bounds BOTH the compressed upload and the total extracted bytes —
zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
escape and device/FIFO members rejected), and atomically swaps the
content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
archive suffix — no missing window, a failed upload never touches the
existing folder/row/KB). The git_sources row is upserted by path
(kind='local', no duplicates, added_at preserved), the models are
checked fail-fast (503 sanitized when down — the folder/row stay
committed and the next sync/re-upload retries idempotently), and the
source is scanned synchronously in the request (single-source
import_sources prune=True + change-gated KB overview), answering 200
with the sync-style counts. One upload at a time (409); the request
session is released before the scan so a concurrent TRUNCATE cannot
deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
archive_source_name (safe-name derivation), unpack_archive (guarded
zip/tar extraction with the extracted-byte cap, no partial state),
swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
~/bor-sources/uploads, deliberately separate from the git checkouts)
and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
removed; #archive-upload-form takes its place (labeled file input,
"Upload & scan" button, the §7.4 never-stale lifecycle, inline
role=alert error, role=status count line); hint + table caption
updated. The POST /api/git-sources kind=local API contract is
UNCHANGED — a plain directory is still registrable via the API, and
existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
upload→scan→list (the deterministic "Uploading…" in-flight state, the
Local row, /api/docs + the RAG catalog), same-filename re-upload
(in-place replace, prune, no duplicate row, v2-only folder), the
422 inline error + recovery (the form is not wedged), and the
anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
replace, both new settings), the local-directory form removal noted,
config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.
Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.
Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.
- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
import_extensions CSV gain the ten names (the original seven first);
the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
(HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
default CSV + dotted import_extension_set include the ten, the
validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
new suffix (parametrized), the .container fixture chunks >=2 under
the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
tree indexes exactly the ten new files (unknown/hidden/excluded
filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
over a temp tree with .container/.volume/.j2 -> documents + chunks
rows with stem titles; delta re-import updates only the changed .j2
doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
isolation): GET /api/docs (admin session) lists the four new-format
docs with non-zero chunk counts and stem titles; the Sources table
renders a row + .doc-link per file; the phase-26 modal shows the
.container TOML ([Container] section + sentinel) with stem title and
the container format badge; a RESE-JINJA-SENTINEL-33dd question
FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
FTS hits) — the bubble is not .is-deflected and a source chip names
templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
the revision note under the anchors table, and the §5 chunking-policy
+ §11 workflow lines. The only PLAN edit this phase.
Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.
Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
Phase 45 (owner permission 2026-08-27, TODO.md L8: "allow the LLM
to make as many tool calls as it wants"): the phase-37 per-turn tool
budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each)
and their exhaustion refusals are removed — a grounded turn now offers
list_documents / read_document for the whole turn (re-lists included),
bounded only by the round cap:
- app/config.py: agent_max_rounds (BOR_AGENT_MAX_ROUNDS, default 10,
negative rejected) replaces agent_list_calls / agent_read_calls;
.env.example + README document the single knob; app/rag/prompts.py
docstrings follow.
- app/rag/agent.py: the loop runs tools until the model answers or
rounds >= max_rounds, at which point it forces one final no-tools
answer (the cap is the only forced exit); 0 = no tools — exactly one
tools=None request, byte-identical to the pre-phase-37 path (the
kill switch). Rejected calls (unknown tool / missing args /
already-in-context / unknown path) still consume a round, so
pathological rejected-call streams are bounded by the cap. The
per-call log line is now tool/args/round=N/M; the per-turn
tool_calls=N field and the tool SSE event are unchanged.
- tests/e2e/mock_llm.py: MULTI_READ_TRIGGER ("read two documents") —
the deterministic list -> read #1 -> read #2 -> forced-answer flow
(byte-stable "I read <sp1> and <sp2>." line), classified by the
count of tool-role read results; the phase-37 single-read flow stays
byte-identical (unit-pinned in tests/unit/test_mock_tool_flow.py).
- tests/e2e/test_agent_unlimited_tools.py (new, story suite,
mock-only): three tool frames/lines in order (one list, two reads —
the second read is what the old read budget refused) + the
both-named non-deflected answer; done.sources + chips = retrieval
doc + both reads, deduped; no budget refusal rendered; the
single-read marker flow regression (exactly one read, single tool
pair).
- .agent/PLAN.md: the phase-45 SSE revision note (owner-locked, R2) —
the only PLAN edit this phase; the phase-37 note's budget clause is
marked removed.
Unit/integration rewrites (test_agent.py round-cap matrix incl. the
kill switch and rejected-call spam, test_config.py, test_chat_api.py
agent_max_rounds=0 fixtures) landed with the server core so every gate
stays green.
uv run pytest: 756 passed, app/ coverage 99%; ruff + pyright clean;
story E2E 4/4 in isolation (ran twice); regression E2E suites
(agent_document_tools unmodified, chat_rag, smoke) green in isolation.
Also records the 45_agent_unlimited_tools todo/ -> complete/ task-file
moves (00/01/02 pending in the working tree, task 03 moves on success).
GFM pipe tables in the shared renderer (TODO.md L6): a table-protection
pass in frontend/assets/markdown.js (fences -> tables -> escape order)
pulls each header+separator+body block out as a placeholder, renders
cells escape-first with the same inline transforms, and reinserts a
semantic <table class="md-table"> inside a horizontal-overflow
.md-table-wrap — so a pipe table in a chat answer, the document
viewer/modal, and the thinking block all render the same semantic
table. Fences win over tables; lone pipes stay text.
- styles.css: .md-table palette rules (PLAN §7.2 tokens, no motion);
min-width: max-content so a WIDE table keeps its natural width and
the wrapper is the real scroller (width:100% alone wrapped the wide
table's cells — proven by the new E2E).
- mock_llm.py: TABLE_TRIGGER ("show me a table") -> byte-stable
TABLE_ANSWER (3-column table, <img onerror> XSS probe line, wide
5-column table), checked before DEFLECT_MODE like SUMMARY_MODE.
- tests/fixtures/docs/homelab/tables.md: 3x3 pipe table + pipe-heavy
fenced block (viewer/fence subject); the shared fixture set grows
8 -> 9 docs, so every suite pinning the count (added/formats/
stat-docs/EXPECTED_ROWS) is updated accordingly.
- tests/e2e/test_markdown_tables.py (new, story suite): chat table
shape + non-deflection, wide-table wrapper scroll (no page
overflow), XSS probe inert, viewer modal table, fence-not-a-table,
lone pipe stays text.
- tests/e2e/test_agent_document_tools.py: fix a pre-existing flake —
the "Calling tool…" label window is ~0.4 s at the mock's 0.1 s
tool-frame pacing, and a polling expect could stride over it
(failed 3 of 5 runs on the committed baseline). The pre-submit
MutationObserver record is the deterministic source of truth; the
racy to_have_text gate is gone.
uv run pytest: 738 passed, app/ coverage 99% (TOTAL unchanged);
ruff + pyright clean; story E2E 6/6 in isolation; regression E2E
suites (chat_rag, document_viewer, document_summaries, smoke) green.
TODO.md L4: with a dead model endpoint the sync discovered it only
mid-import, after slow clones — and a tooltip on the button is not a
readable error.
- app/rag/llm.py: ModelUnavailableError + check_models(llm) — a tiny
pre-sync probe (one short embedding + one 1-token-scale completion)
that fails naming the unavailable model (embed first, then the
summary model); the sync sanitizer still masks credentials.
- app/api/sync.py: the probe is step 1 of _run_sync — before source
resolution and before any clone_or_pull; a model failure is just
another 'failed' state (no new endpoint, A10/A12 untouched).
- frontend/assets/header.js: applySyncFailure now also opens the
module-owned error modal (every page carrying #sync-btn, zero
page-markup changes): lazily built backdrop + role=alertdialog
panel, error text via textContent, close via button / Esc /
backdrop, focus in-and-out to #sync-btn (with a body→#sync-btn
fallback — the run's disabled button drops focus to <body>).
- frontend/assets/styles.css: the modal on the phase-08 error palette
(z-index above the header, .is-open open/close, reduced-motion
stilling, 44px close target).
- Tests: probe unit tests (both up / embed down / summary down /
custom model names), sync integration (fail-fast before any clone,
probe-before-effective_sources ordering, credential masking,
healthy regression), the phase-41 source pins, and the story E2E
(two module apps on distinct ports — dead endpoint on a closed
loopback port vs session mock: ≤10 s fail-fast + modal contract,
all three dismissal paths with focus out to #sync-btn, button
title/.is-error + Sources banner untouched, healthy phase-32
lifecycle regression to 'Synced HH:MM').
E2E (isolation): test_sync_model_down.py 4/4, test_sync_button.py
3/3, test_git_sources_admin.py 6/6, test_local_directory_sources.py
3/3; unit+integration 721 passed, app/ coverage 99%; ruff + pyright
clean.
One env var (BOR_APP_NAME, default "Brain of Reese") now drives the app's
display name everywhere (TODO.md L12 — owner ask: "a way to customize the
name for 'Brain of'. Should be an env var."). The existing app_name setting
is the source of truth (phase locked decision — no new variable, no rename);
with the variable unset the app is byte-identical to before.
Endpoint (A10 public/stateless, no secrets):
GET /api/config → exactly {app_name, version} (app/api/config.py, the
health.py pattern; registered before the static mount). Integration tests:
anonymous 200, default values, a Settings override follows, key set is
exactly two keys — no other setting may leak in later.
Frontend brand layer (A11 — runtime fetch, static templates stay static):
assets/brand.js — a CLASSIC script, first on all six pages, so its top
level runs at parse time: window.BOR_BRAND = "Brain of Reese"
synchronously (the default renders immediately, no blank flash), then a
no-store fetch of /api/config applies the name — document.title (global
replace), every .brand-text (a name starting "Brain of " keeps the bold
split Brain of <strong>rest</strong>, any other name renders plain; the
operator-controlled name is HTML-escaped before innerHTML), a TreeWalker
over text nodes (script/style rejected — page source never rewritten),
and the aria-label/placeholder/meta-content attributes. Fetch failure
keeps the default + console.warn (the loadHealth house style).
app.js (status labels, typing label, elapsed-hint aria, tool labels) and
document.js (viewer titles) read window.BOR_BRAND at CALL time via
brand() — a label set after the fetch lands carries the configured name.
Containerfile: esbuild minify line for brand.js (classic, like markdown.js);
the phase-33 ?v= cache-busting picks the new asset ref up automatically.
E2E (A16 — one story, one file, isolated): test_configurable_brand.py boots
a SECOND app instance (same DB/mock-LLM/admin-auth env block, port APP_PORT+1,
BOR_APP_NAME="Brain of Testy") — the shared conftest server keeps the
default name so every other suite's title/label assertions stay untouched —
and asserts /api/config on both instances, the index title/brand/greeting/
#messages aria-label, the sources + login page titles, and one pre-token
chat turn (think out loud marker) whose #send-status reads "Brain of Testy
is thinking"; the no-op regression pins the shared server's default bytes.
Docs: .env.example App section + README configuration reference — what it
affects (titles, header brand, status labels, aria text), the default, the
bold-split rendering rule.
Gates: 695 unit+integration passed, app/ coverage 99% (>90%), story E2E
green in isolation (two consecutive runs), brand-string suites (smoke,
shared header, header consistency, chat persistence) green, ruff + pyright
clean.
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
Grounded chat turns now run the agent loop (app/rag/agent.py) instead
of a bare chat_stream: while the per-turn budgets last
(BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each) the model
gets list_documents (the indexed catalog, /api/docs order) and
read_document (full text, never truncated — A7-revised contract); once
both budgets are spent the tools key is dropped from the request and
the model must answer. Rejected calls (unknown tool, unknown/missing
path, document already in context, spent budget) consume no budget.
Budgets 0/0 make exactly one tools=None request — byte-identical to
the pre-phase path (budgets-as-kill-switch). Deflected turns keep the
direct chat_stream (A8 unchanged; the LOW prompt never carries the
<tools> section).
SSE contract gains {"type":"tool","name":...,"argument":
"source/path"|null} frames ahead of the answer deltas (PLAN §4
extension, owner permission 2026-08-26); done.sources, query_log.sources
and the per-turn log line (gains tool_calls=N) report the retrieval
docs + read docs, deduped. The UI shows a "calling tool"
button/label state and one visible .tool-call line per call above the
answer; the lines persist with the chat record and re-render on
reload. chat_stream passes tools through and accumulates streaming
tool_calls deltas into ToolCallPiece (tools=None stays byte-identical).
E2E: deterministic mock tool flow ("use your tools" + <tools> marker:
list -> read first catalog line -> quoted answer) plus the story suite
(marker flow, reload re-render, plain/deflected no-tool regressions).
Docs: .env.example + README (the two tools, the budgets, the SSE tool
frame, the "calling tool" UI state).
probe: turbo tool_calls=supported 2026-08-26 (uv run python -m
scripts.llm_probe --tools — non-streaming + streaming
finish_reason=tool_calls, indexed delta.tool_calls partials)
Phase 33 (story: .agent/user_stories/cache-busting.md).
- app/core/caching.py: asset_version() — git short SHA (a commit is a
deploy), stable content-hash fallback for non-git checkouts, "dev"
for a missing static dir; computed once per process. CachingMiddleware
— the five HTML pages revalidate (no-cache) with ?v=<token> asset refs
rewritten in flight; /assets/* is public, max-age=31536000, immutable;
everything else (all /api/*, the SSE chat stream in particular) passes
through byte-identical.
- tests/e2e/test_cache_busting.py: fresh-Chromium wire assertions —
document no-cache, versioned CSS/JS request URLs sharing one token,
immutable asset headers, /api/health baseline headers, SSE chat to
done (mock LLM).
- README 'Caching / deploys' section + story file.
Also fixed two prod-image defects surfaced by this phase's podman smoke
(the full app would not boot):
- Containerfile: ship the scripts/ package — app/api/sync.py (phase 32)
imports scripts.git_sync / scripts.import_docs at module level, so the
container crashed on boot (ModuleNotFoundError: No module named
'scripts').
- compose.yaml: pass BOR_ADMIN_PASSWORD / BOR_SESSION_SECRET through to
the app service (:- defaults keep 'podman compose up -d db' working;
the app's own fail-loud gate still names missing admin auth).
Smoke: podman compose --profile prod up -d on a fresh image + a fresh
Chromium profile — /, /sources.html and /login.html all served
Cache-Control: no-cache; all 8 asset requests versioned with one shared
token (content-hash fallback inside the image — no .git there);
/assets/* immutable for a year.
ducoterra