Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(agents): migrate .agent/ planning tree to .agents/

Browse Source 
Standardize on the .agents/ directory (shared with project skills):
phases/, user_stories/, reports/, screenshots/, validate.sh, and
phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves
history; runtime artifacts move alongside).

Updates every reference in AGENTS.md, README.md, .gitignore, app
docstrings, and test story headers. Historical KB content in data/
and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
ducoterra
2026-09-05 10:57:07 -04:00
parent 766702c750
commit dbf2af26c6
1118 changed files with 664 additions and 664 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__01_settings_and_unpack_utility.a1.err
View File
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__01_settings_and_unpack_utility.a1.md
+11
View File
@@ -0,0 +1,11 @@
All green and scoped. Those other modified files are pre-existing uncommitted work (owner 2026-08-28 rework from earlier phases), untouched by me.
**Task 01 complete — settings + safe archive unpack utility**
- `app/config.py`: `upload_dir` (`~/bor-sources/uploads`, `BOR_UPLOAD_DIR`) + `upload_max_mb` (512, `BOR_UPLOAD_MAX_MB`) with fail-loud `<= 0` validator; `.env.example` documents both beside `BOR_SOURCES_DIR`.
- `app/rag/archive_upload.py` (new, no FastAPI/DB): `ArchiveUploadError`, longest-first `ARCHIVE_SUFFIXES`, `archive_source_name` (case-insensitive suffix strip, stem case preserved, rejects separators/control chars/`.`/`..`/empty + bare-suffix stems like `tar.gz`), `unpack_archive` (content-sniffed zip/tar; rejects absolute/`..`/escape-resolving members, zip symlink + non-regular entries, tar device/FIFO, escaping symlink/hardlink targets; byte-capped streaming; full cleanup of partial target on any failure), `swap_in` (rename to `.old-<hex>` sibling → rename in → delete; best-effort restore + re-raise on failure).
- `tests/unit/test_archive_upload.py` (new, 61 tests): full derivation matrix, byte-identical valid zip/tar.gz/plain-tar extracts, every guard raising with no partial target, cap exact/accumulate/corrupt cases, swap fresh/replace/restore-on-failure/double-failure, settings defaults/overrides/validator.
- Tests: `uv run pytest tests/unit/test_archive_upload.py -v` → 61 passed; `uv run pytest tests/unit tests/integration` → 855 passed; `uv run pytest --cov=app` → new module 100%, TOTAL 99% (>90%).
- Lint: `uv run ruff check .` clean; `uv run pyright` → 0 errors.
- Notable: CPython `writestr`/Windows zips carry no file-type mode bits — only modes with explicit non-regular type bits are rejected (name-decided otherwise); bare `tar.gz` (no dot) treated as empty stem per task matrix. No existing files touched beyond config + `.env.example`.
- Next pending task: `.agents/phases/todo/49_archive_upload_sources/02_upload_api.md`.
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__01_settings_and_unpack_utility.a1.validate
+62
View File
@@ -0,0 +1,62 @@
........................................................................ [ 8%]
........................................................................ [ 16%]
........................................................................ [ 25%]
........................................................................ [ 33%]
........................................................................ [ 42%]
........................................................................ [ 50%]
........................................................................ [ 58%]
........................................................................ [ 67%]
........................................................................ [ 75%]
........................................................................ [ 84%]
........................................................................ [ 92%]
............................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 122 0 100%
app/api/config.py 7 0 100%
app/api/docs.py 23 0 100%
app/api/git_sources.py 66 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 81 0 100%
app/config.py 79 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 92 0 100%
app/core/debugging.py 29 2 93%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 44 0 100%
app/models.py 60 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 72 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 186 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 99 0 100%
-----------------------------------------------
TOTAL 1899 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__02_upload_api.a1.err
View File
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__02_upload_api.a1.md
+21
View File
@@ -0,0 +1,21 @@
All green. Task 02 complete.
**Implemented**
- `pyproject.toml`: added `python-multipart>=0.0.9,<0.1` (FastAPI multipart parser); `uv lock` + `uv sync` clean (0.0.32)
- `app/schemas.py`: new `UploadOut` (`source` + `files/added/updated/unchanged/pruned/errors/chunks` + `overview`)
- `app/api/git_sources.py`: `POST /api/git-sources/upload` — format/name gate (422 naming accepted set), module-level in-flight flag (409, phase-32 spirit; loop-independent), 1 MiB-chunk stream with `upload_max_mb` cap (413 + temp cleanup), safe unpack to temp sibling (422, zero-entry = "the archive contains no files"), atomic `swap_in`, row upsert by `path` (existing row/`added_at` preserved; `_commit_new` 409 backstop), fail-fast `check_models` (503, `_sanitize_error` from `app.api.sync`), `import_sources(prune=True)` + change-gated `regenerate_overview`, per-upload INFO log line, 200 with sync-style counts. Existing GET/POST/DELETE handlers byte-identical (diff = docstring/imports/new route only)
- `tests/integration/test_git_sources_upload.py`: 21 tests — anonymous 403; 422 formats/unsafe names; 409 (real in-flight via thread + flag seam); 413 + exact-cap boundary + no stray temps; zip-slip/tar-symlink/corrupt/swap-failure all 422 with previous folder+row+KB asserted intact; zero-entry 422; happy paths (`.tar.gz`/`.zip`/`.tar`/`.tgz`) incl. log line + `kind=local` row; re-upload in-place replace (one row, `added_at` unchanged, 2 pruned); non-A9 replace (valid, no overview); 503 sanitized with folder/row committed
**Results**
- `uv run pytest tests/integration/test_git_sources_upload.py -v` → 21 passed
- `uv run pytest tests/unit tests/integration` → all passed (exit 0; 76 passed across upload + `test_git_sources_api` + `test_sync_api` + `test_import_docs_git`)
- `uv run pytest --cov=app --cov-report=term-missing` → exit 0, **TOTAL 99%** (>90%); `app/api/git_sources.py` 100%
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
**Decisions/deviations**
- Plain bool flag instead of `asyncio.Lock` (task-allowed; TestClient runs separate loops per request)
- Tests post multipart via httpx `files=` — with this httpx (0.28.1), a tuple in `data=` is sent urlencoded and never parses as `UploadFile`
- Control-char filenames are percent-encoded by the multipart transport before reaching the app (verified live); that branch stays unit-covered in `test_archive_upload.py`
- Bare-suffix names (`tar.gz`) pass the format gate and get task-01's "no usable source name" 422 (more precise than the formats message)
**Next pending task:** `.agents/phases/todo/49_archive_upload_sources/03_admin_page_upload.md`
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__02_upload_api.a1.validate
+63
View File
@@ -0,0 +1,63 @@
........................................................................ [ 8%]
........................................................................ [ 16%]
........................................................................ [ 24%]
........................................................................ [ 32%]
........................................................................ [ 41%]
........................................................................ [ 49%]
........................................................................ [ 57%]
........................................................................ [ 65%]
........................................................................ [ 73%]
........................................................................ [ 82%]
........................................................................ [ 90%]
........................................................................ [ 98%]
............ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 122 0 100%
app/api/config.py 7 0 100%
app/api/docs.py 23 0 100%
app/api/git_sources.py 142 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 81 0 100%
app/config.py 79 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 92 0 100%
app/core/debugging.py 29 2 93%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 44 0 100%
app/models.py 60 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 72 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 186 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 109 0 100%
-----------------------------------------------
TOTAL 1985 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__03_admin_page_upload.a1.err
View File
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__03_admin_page_upload.a1.md
+20
View File
@@ -0,0 +1,20 @@
All green. Final report:
## Task 03 complete — Sources page: local form out, upload form in (+ phase-38 E2E rewrite)
**Implemented**
- `git-sources.html`: `#local-source-form` removed; `#archive-upload-form` added (file input w/ accept set, never-stale button, `role=alert` error, `role=status` result line); hint + table caption updated; `<title>`/`<h1>` untouched.
- `styles.css`: upload form card/button/file-input/result-line rules (existing palette, ≥44px, global 3px `:focus-visible`), mobile stack block updated.
- `git-sources.js`: local-form refs + second `wireAddForm` removed; upload wiring (§7.4 lifecycle, FormData without manual Content-Type, sync-style counts via `fmtUploadResult` per `sources.js` convention, announce, `loadSources()`, file kept on failure).
- `test_local_directory_sources.py` rewritten fully API-driven (phase-49 note in docstring); acceptance (Local badge, add/remove, Sync import/prune, gate/403s incl. upload route) preserved.
- **Defect fixed (task 02 work):** upload handler held the request `db` session open across the scan → concurrent E2E `TRUNCATE` deadlocked (invisible 3-connection cycle, app + pytest hung forever; reproduced via py-spy/pg_locks). Now `db.close()` after the row upsert; regression test added (verified it fails without the fix: `LockNotAvailable`).
**Results**
- `uv run pytest --cov=app --cov-report=term-missing` → 877 passed, TOTAL 99% (>90%).
- `uv run ruff check . && uv run pyright` → clean.
- `uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov` → 3 passed in isolation; regressions green in isolation: `test_git_sources_admin` (6), `test_nav_rename_sources` (4), `test_smoke` (3), `test_sync_button` (3), `test_import_documents` (3); no orphan processes.
- Upload form verified live end-to-end via a throwaway E2E (deleted after): "3 added" → re-upload "1 added · 1 updated · 2 pruned", one row, 422 inline w/ file kept.
**Deviation:** task snippet `new FormData([["file", file]])` isn't a valid browser API — used `new FormData(uploadFormEl)` (same multipart contract, field `file`).
**Next pending task:** `.agents/phases/todo/49_archive_upload_sources/04_story_e2e_docs_commit.md`
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__03_admin_page_upload.a1.validate
+63
View File
@@ -0,0 +1,63 @@
........................................................................ [ 8%]
........................................................................ [ 16%]
........................................................................ [ 24%]
........................................................................ [ 32%]
........................................................................ [ 41%]
........................................................................ [ 49%]
........................................................................ [ 57%]
........................................................................ [ 65%]
........................................................................ [ 73%]
........................................................................ [ 82%]
........................................................................ [ 90%]
........................................................................ [ 98%]
............. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 122 0 100%
app/api/config.py 7 0 100%
app/api/docs.py 23 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 81 0 100%
app/config.py 79 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 92 0 100%
app/core/debugging.py 29 2 93%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 44 0 100%
app/models.py 60 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 72 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 186 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 109 0 100%
-----------------------------------------------
TOTAL 1986 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__04_story_e2e_docs_commit.a1.err
View File
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__04_story_e2e_docs_commit.a1.md
+10
View File
@@ -0,0 +1,10 @@
All gates green. Phase 49 is complete.
**Report**
- **Story E2E**: created `tests/e2e/test_archive_upload_sources.py` (5 tests: form swap, upload→scan→list with deterministic "Uploading…" in-flight state via request-hold, same-filename in-place re-upload with prune + no duplicate row + v2-only folder, 422 inline error + recovery, anonymous gate + 403); real `tarfile` archives `e2e-upload.tar.gz` v1/v2 with ALPHA/BETA/GAMMA sentinels; `BOR_UPLOAD_DIR` pointed at a host-inspectable scratch dir.
- **README**: new *Archive upload sources* section (formats, naming rule, in-place replace + prune, `BOR_UPLOAD_DIR`/`BOR_UPLOAD_MAX_MB`, synchronous scan semantics), local-directory form removal + API-only note, UI bullet, sync note, two config-table rows (`.env.example` already documented both settings in task 01).
- **Tests/lint/coverage**: `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` → 5 passed (isolated, DB up); regression suites isolated → git_sources_admin 6, local_directory_sources 3, sync_button 3, import_documents 3, nav_rename_sources 4, smoke 3, shared_header 6 — all passed; `uv run pytest` → 877 passed; `--cov=app` → **99%** (>90%); `ruff check .` + `pyright` clean.
- **Commit**: `03d2625 feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place` (`--no-gpg-sign`) — exactly the 21 phase files incl. story file + force-added moved phase dir; `todo/` no longer lists 49.
- **Notes**: one pyright fix in my new test (`text_content()` None guard); a small pre-existing in-flight hunk (`.sign-in-mobile` CSS) rode in via `styles.css`, disclosed in the commit note per the phase-48 precedent; pre-existing uncommitted in-flight rework (sources.js sync-button etc.) and old `todo/40–47` tracked deletions were left untouched — out of this phase's file list.
- **Next pending task**: none — `todo/` is empty; phase 49 was the last phase.
.agents/reports/49_archive_upload_sources/49_archive_upload_sources__04_story_e2e_docs_commit.a1.validate
+63
View File
@@ -0,0 +1,63 @@
........................................................................ [ 8%]
........................................................................ [ 16%]
........................................................................ [ 24%]
........................................................................ [ 32%]
........................................................................ [ 41%]
........................................................................ [ 49%]
........................................................................ [ 57%]
........................................................................ [ 65%]
........................................................................ [ 73%]
........................................................................ [ 82%]
........................................................................ [ 90%]
........................................................................ [ 98%]
............. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 122 0 100%
app/api/config.py 7 0 100%
app/api/docs.py 23 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 81 0 100%
app/config.py 79 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 92 0 100%
app/core/debugging.py 29 2 93%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 44 0 100%
app/models.py 60 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 72 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 186 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 109 0 100%
-----------------------------------------------
TOTAL 1986 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK