6e654df387bd859ad02a9ad5918bad858631cd28
58
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ducoterra
|
6e654df387 |
phase: 127_sync_incremental_folder_summaries
All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact. **Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass) - **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors** - **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me) - **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed - **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit) - **Next pending phase:** `128_sync_label_mobile` |
||
|
ducoterra
|
e0f11e7aa8 |
phase: 126_pdf_support
All gates verified green — no code changes were needed on this pass. Final report: ## Phase 126 — PDF support: final verification pass **Verified (all tasks 01–05 already complete in working tree):** - `pymupdf` dep in `pyproject.toml`/`uv.lock`; `.env.example` documents the `pdf` token - `app/rag/pdf.py` (text + image extraction, MIME mapping), page-aware `_chunk_pdf`, importer `_index_pdf_file` + embedded-image pass, archives covered via extension walk **Test / lint / coverage results:** - `uv run pytest` → 2997 passed - `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` (isolation) → 6 passed - `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate), 2997 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** 1. PDF → doc with text/chunks/embedding/summary — ✅ (integration + E2E Sources-page tests) 2. `BOR_IMAGES=true`: embedded images → standalone image docs — ✅ (integration + E2E chat-chip test) 3. `BOR_IMAGES=false`: images skipped, text indexed — ✅ (both suites) 4. PDF renders as extracted text in viewer — ✅ (E2E) 5. Test/coverage/lint gates — ✅ (above) 6. Commit + phase move — harness handles (per protocol, no `git` on this pass) **Decisions:** none new; inline `[Embedded image: …]` placeholders were a documented task-04 deviation (simplified spec wins) — not a defect. No defects found; repo left functional with changes in the working tree. **Next pending phase:** none — `todo/` is empty once 126 is moved. |
||
|
ducoterra
|
e0307fe3ff |
phase: 125_image_anchor_context
All completion criteria verified. The repository is left functional with all changes in the working tree (no commits made, per the harness protocol). ## Final report — phase 125 `image_anchor_context` (verification pass) All 6 tasks were already complete; this pass verified every criterion and ran the full gates: - **Migration 0023** `documents.image_context` (TEXT nullable, reversible) — applied (`alembic current` = 0023 head); round-trip test (4 passed) exercises real upgrade → downgrade 0022 → upgrade. - **Criterion 1** (referenced image enriched + context line in retrieval): integration `test_markdown_referenced_image_gets_context_and_enriched_chunks` + E2E `test_sync_enriches_the_image_doc_with_markdown_context`, `test_chat_answer_quotes_the_context_line` (wire prompt pinned: `<document>` block leads with the one-line `Referenced in …: "alt"` line) — verified present and passing. - **Criterion 2** (standalone image byte-identical): E2E `test_standalone_*` (NULL `image_context`, block body exactly the bare description, no `Referenced in` anywhere in the prompt) + unit `test_describe_image_without_context_is_byte_identical` — passing. - **Criterion 3** (markdown-first: record stored on markdown doc, image picks up context later): integration `test_markdown_indexed_before_image_stores_record_and_image_picks_it_up` + post-index enrichment pass test (markdown file gone from disk) — passing. - **Gates:** `uv run pytest` → 2938 passed, exit 0 · `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL **99%** (>90%) · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors, 0 warnings. - **Phase E2E in isolation:** `uv run pytest tests/e2e/test_image_anchor_context.py -v --no-cov` → 4 passed. - **Mock-LLM change regression check:** neighboring image E2E suites green in isolation — `test_smoke` (3), `test_image_documents` (16), `test_chat_image_questions` (17), `test_answer_images` (17). - No defects found; no code changes needed. No live-infrastructure changes (local dev DB only). No commit made (harness commits). - Next pending phase: check `.agents/phases/todo/` after the harness moves this phase. |
||
|
ducoterra
|
62e4f6b572 |
phase: 124_image_chips_and_answers
All green. Final report: **Phase 124 final verification pass — all criteria verified, no defects found (nothing to fix)** - Verified the implemented phase (all 7 tasks done, changes in working tree): `BOR_ANSWER_IMAGES` toggle, `show_image` tool + refusals + `holder.shown_images`, toggle-gated prompt clause, `answer_image` frame field (None-omitting serializer) + `ChatMessage` persistence, `merge_cited_docs` auto-cite bug fix, answer-figure UI (live/restore/shared/CSS), mock triggers + E2E suite. Also confirmed the bytes route → public change matches the phase-123 shared-chat precedent (task 06 design). - `uv run pytest --cov=app --cov-report=term-missing` → **2848 passed**, TOTAL **99%** (>90% gate) - `bash .agents/validate.sh` → **validation OK** (tests + coverage + ruff + pyright) - `uv run ruff check . && uv run pyright` → clean, 0 errors - `uv run pytest tests/e2e/test_answer_images.py -v --no-cov` (isolated) → **5/5 passed** - Mock-inert spot checks: `tests/e2e/test_image_documents.py` 4/4, `tests/e2e/test_agent_document_tools.py` 4/4 **Completion criteria:** ① zero-read suggested-tier image doc chips with figure — verified (unit matrix, integration `test_zero_read_grounded_turn_auto_cites_suggested_image_doc`, E2E bug-fix flows); ② show_image → answer figure below text/above chips, second call refused, persists + restore + shared — verified (unit handler matrix, integration round-trip, E2E feature flow); ③ default-off byte-identical (3 tools, no clause, no key) — verified (unit + integration + E2E toggle-off); ④ full suite green / >90% / lint clean — verified above; ⑤ commit + phase move — harness responsibility (changes left uncommitted in working tree, no live-infrastructure changes made). **Next pending phase:** none — `todo/` holds only this phase; the pipeline is clear once the harness moves it. |
||
|
ducoterra
|
bef24e05e2 |
phase: 123_chat_image_questions
All gates green. Verification complete. **Phase 123 — final verification pass (all 4 tasks already in `complete/`)** - Verified the full implementation is in the working tree: `app/api/chat_images.py` (upload/serve pair), `ChatRequest.image`/`ChatMessage.image` (path-validated, omitted-when-None), toggle-off + stale-file hinted error frames, `build_user_content` multimodal build at both sites (chat.py deflected branch + `run_agent`), config-gated composer attach/preview/upload-then-send, restore + shared rendering, CSP `img-src 'self' data:` carve-out, mock-LLM capture buffer. - `uv run pytest` → **2796 passed**, exit 0 (unit + integration). - `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (29/4615 missed; phase-123 modules 99–100%). - `uv run pytest tests/e2e/test_chat_image_questions.py -v --no-cov` → **5 passed** in isolation. - `uv run ruff check . && uv run pyright` → clean (0 errors). **Completion criteria:** (1) attach→send→multimodal text+image to the model, bubble/reload/shared all render it, saved chat stores the PATH with `"base64" not in json.dumps(stored)` — **verified** (E2E tests 1–4 + integration round-trip); (2) `BOR_IMAGES=false` — control hidden, exact hinted error frame, zero model calls / no query_log row — **verified** (E2E test 5 + integration); (3) text-only byte-identical (`content` stays a plain `str`) — **verified** (unit + integration); (4) all gates green — **verified**; (5) commit + phase move — left to the harness per pipeline rules (no `git add`/`commit` run). No defects found; no live-infrastructure changes (repo + local dev DB only). **Next pending phase: none** — 123 is the last phase in `todo/`. |
||
|
ducoterra
|
a19d78d284 |
phase: 122_image_documents
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**
**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs
**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)
**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).
**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.
**Next pending phase:** `123_chat_image_questions`.
|
||
|
ducoterra
|
0f77e9a876 |
phase: 121_git_source_tokens
**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed) - Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes - Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate) - Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed** Completion criteria: 1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`) 2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking) 3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests) 4. pytest / coverage / ruff / pyright — **PASS** (see above) 5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol) Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions). |
||
|
ducoterra
|
a5b63f83ad |
phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119. |
||
|
ducoterra
|
9820c361b0 |
phase: 118_summary_seed_context
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.
- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
|
||
|
ducoterra
|
3846f26a58 |
phase: 114_embed_question_length
All verification passes complete — the phase was already fully implemented in the working tree, and every gate is green. No defects found; no code changes were needed.
**Final verification pass — Phase 114 (embed question length):**
- Verified truncation: `chat.py:459` embeds `request.message[:settings.embed_question_max_chars]` (default 1200, `BOR_EMBED_QUESTION_MAX_CHARS`, `>0` validator); full question still reaches the LLM prompt/log.
- Verified error mapping: `EmbeddingInputTooLargeError(EmbeddingError)` (byte-identical message) caught **before** `EmbeddingError` → no retry, terminal frame `detail="Question too long — trim it and re-ask."` + reachability-fine hint; `ChatErrorEvent.hint` additive.
- Verified frontend chain: frame `hint` → `err.hint` → `setUiState(error, …, {hint})` → `showErrorBanner(…, {retryable: true})` — hint replaces default `ERROR_HINT`, phase-111 `#banner-retry` button revealed. 4,000-char clamp untouched (A2).
- `uv run pytest tests/unit/test_embed_question_length.py -v --no-cov` → 21 passed
- `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (isolation, DB up) → 1 passed (4,000-char question → done, no banner)
- Regression: `test_llm_retry.py` 4 passed · `test_oneshot_llm_retry.py` 2 passed · `test_chip_sizing_question_cap.py` 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2444 passed, TOTAL **99%** (>90% gate)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
**Completion criteria:** (1) 4,000-char question embeds prefix + full prompt ✅ · (2) too-large → accurate frame + hint + Retry button ✅ · (3) reachability failure byte-identical (retries + old copy) ✅ · (4) all gates green ✅ · (5) commit/phase-move → left to the harness per instructions (no `git add`/`commit` run).
**Deviations:** none. **Next pending phase:** `115_doc_draft_discard`.
|
||
|
ducoterra
|
97d663d16d |
phase: 113_source_chip_quality
All gates green — no defects found; this pass was verification only. **Phase 113 final verification pass — report** - Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries - `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate) - `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed - Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed - `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK" Completion criteria: 1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed 2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed 3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed 4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed 5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`) No deviations. Next pending phase: `114_embed_question_length`. |
||
|
ducoterra
|
1374faf136 |
phase: 112_honesty_gate_weak_hits
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):** - Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`. - Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length). - Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented. - Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW). - E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**. - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors. **Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree). **Deviations:** none. Next pending phase: `113_source_chip_quality`. |
||
|
ducoterra
|
3a4035fc96 |
phase: 110_fix_sse_db_pool_exhaustion
--- **Phase 110 — Fix SSE DB Connection Pool Exhaustion (SEC-14-04): COMPLETE** **What was implemented/verified:** - All three tasks (pool config, short-lived sessions, concurrency cap) were already implemented in code - Fixed `tests/integration/test_chat_db_sessions.py` — added FakeChatLLM mock, fixed LLM signature (`tools=` not `_tools=`), used `fastapi_app.dependency_overrides` instead of `client.app.dependency_overrides` - Fixed `tests/e2e/test_chat_db_pool.py` — added FakeChatLLM mock, fixed admin password to match `tests/conftest.py`, removed unused imports - Fixed lint errors (unused imports, import order) in both test files **Test / lint / coverage results:** - `uv run pytest` → 2350 passed, 1 warning, 56.4s - `uv run pytest --cov=app --cov-report=term-missing` → 99% coverage (4065 lines, 16 uncovered) - `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov` → 3 passed - `uv run pytest tests/integration/test_chat_db_sessions.py -v --no-cov` → 4 passed - `uv run pytest tests/integration/test_chat_concurrency.py -v --no-cov` → 11 passed - `uv run pytest tests/unit/test_db_pool_config.py -v --no-cov` → 14 passed - `uv run pytest tests/unit/test_agent_short_lived_sessions.py -v --no-cov` → 7 passed - `uv run ruff check .` → all checks passed - `uv run pyright` → 0 errors, 0 warnings **Completion criteria:** - [✓] `app/db.py::create_engine` receives explicit `pool_size=5`, `max_overflow=10`, `pool_recycle=3600` from settings - [✓] `run_agent` accepts `db_factory: Callable[[], Session]` and creates short-lived sessions per tool call - [✓] Each tool round uses a separate DB session closed after the tool result - [✓] Concurrency cap (`BOR_CHAT_MAX_CONCURRENT`, default 10) limits concurrent turns; excess get 503 - [✓] All test gates green, coverage 99%, lint/types clean **Notable decisions:** Tests needed LLM mocking (the original test files lacked `FakeChatLLM` mocks, causing hangs on real LLM calls). **Next pending phase:** None — this is the last phase in `todo/`. |
||
|
ducoterra
|
35d65d2f25 |
feat(rag): summarize single-document folders (MIN_DOCS_PER_FOLDER 2 → 1)
Relax the phase-94 folder-summary scope rule from ≥ 2 documents to ≥ 1: a folder (or source root) is a candidate while ANY document lives under it, so single-file folders and single-file source roots get their own lite-written description. A row is now pruned only when its folder loses its last document (vanishes from the catalogue). The constant is the single source of truth, so the flip propagates to the generator's candidate set, the prune pass, the missing_folder_summaries gap probe (the next sync self-heals the new gaps), and the KB-tree summary_pending markers (1-doc folders / sources now read "Summary pending" until their row lands). Docstrings/comments across app/, scripts/import_docs.py, and the E2E fixtures updated to the ≥ 1 wording. Unit + integration tests updated to the new semantics (the pruned-below-minimum scenario is now a folder losing its LAST doc; single-doc folders are pinned as candidates/pending). Full suite: 2314 passed, app coverage 99%; ruff + pyright clean; folder-summary E2E stories pass in isolation (ls_tree_drilldown, sync_summary_visibility, kb_tree, kb_tree_nav, document_dates, oneshot_llm_retry). |
||
|
ducoterra
|
ee3efb28c9 |
phase: 106_document_dates
Everything is verified green. Final report: **Phase 106 — Document dates (final verification pass; all 10 tasks already complete)** - Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed. - `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓) - `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up) - 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation** - `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings** **Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions. - **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring). - **Next pending phase:** none — `todo/` holds only this phase. |
||
|
ducoterra
|
f665a83b1a |
phase: 98_sync_summary_visibility
All verification complete. Final report: **Phase 98 — Sync summary visibility: final verification pass** (all 5 tasks already complete; implementation verified against the design, no defects found, no code changes needed) - **Implementation checked:** `SyncStatus` phase machine (4 new keys, terminal-keep counts), `on_progress` hook in `generate_folder_summaries`, `summary_pending` on `KbTreeSource`/`KbTreeFolder` + D3 rule in `build_kb_tree`, phase-aware sync labels + pending UI in `sources.js`, `.kb-summary-pending` CSS — all match decisions D1–D5. - **Unit + integration:** `uv run pytest` → 2184 tests, 0 failed/errors (exit 0) - **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (criterion >90% ✓; `app/api/sync.py` and `app/rag/folder_summaries.py` at 100%) - **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings - **Phase E2E (isolation):** `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` → **3 passed** (phase machine, live label, pending markers + gap-fill self-heal) - **Regression suites (each isolated, `--no-cov`):** test_kb_tree ✓, test_ls_tree_drilldown 3 ✓, test_sync_button 3 ✓, test_sync_upload_progress 4 ✓, test_oneshot_llm_retry 2 ✓, test_local_directory_sources 3 ✓ - **Completion criteria:** all 7 verified green — status phase fields + terminal semantics; `Writing KB overview…`/`Summarizing folders… (n/m)` labels (title + aria-live); pending set == `missing_folder_summaries` (integration cross-check pinned at `test_docs_api.py:428`); CLI/`ls` byte-identity (no changes to those paths, pins green); suite/coverage/lint gates; dedicated + regression E2E. Commit left to the harness per protocol (no `git add`/`commit` run). - **Decisions/deviations:** none — no fixes were required this pass. - **Next pending phase:** `99_kb_tree_table_and_back_nav`. |
||
|
ducoterra
|
909c96c7bc
|
minor fix to prevent null bytes from breaking sync | ||
|
ducoterra
|
d731169b8b |
phase: 105_hidden_folders_toggle
All completion criteria verified. Everything is green. **Phase 105 final verification pass — all criteria verified** - Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page - Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified) - Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓ - Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation) - Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed - Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings) **Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched). **Deviations:** none — no defects found; no code changes were needed on this pass. **Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`. |
||
|
ducoterra
|
3b2dea5685 |
phase: 102_extensionless_filenames
All verification complete — every gate green, no defects found in previously completed work. **Phase 102 final verification pass — report** Verified (all three task files present in `complete/`; working-tree implementation matches D1–D5 design): - `match_extension` choke point in `app/rag/importer.py` (walk + `formats` counter), `doc_format` name-token badge in `app/api/docs.py`, config/`.env.example` docs, fixture `tests/fixtures/extensionless_kb/`, integration + E2E suites — all present and correct - Completion criteria: end-to-end sync (✓ integration + E2E), case matrix incl. `mydockerfile`/`Dockerfile.dev`/`.dockerfile` exclusions (✓ unit), `formats=dockerfile:1` not `unknown` (✓ log-line assertion), badge `dockerfile`/`containerfile` + `text` fallback + suffixed unchanged (✓ unit/integration/E2E), prune-on-token-removal (✓ `pruned==2`), suffixed-path rule byte-identical (✓ single-line swap, existing cases untouched) Test / lint results (exact commands): - `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, **99%** coverage (>90% gate) - `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed, isolated, DB up - Regressions isolated: `test_import_documents` 3✓, `test_import_extensions_env` 2✓, `test_quadlet_jinja_import` 4✓, `test_document_viewer` 7✓, `test_kb_tree` 8✓ - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings Notable: commit intentionally not made (harness commits the phase); 102's task files already sit in `complete/`, overview stays in `todo/` for the harness. Next pending phases: 98, 99, 103, 104, 105 (numeric next after 102: `103_suggestions_session_openers`). |
||
|
ducoterra
|
ad7585d474 |
phase: 97_kb_tree_catalog
All completion criteria verified — everything is green, no defects found. Final report: ## Phase 97 final verification pass — ALL GREEN **Verified (no code changes needed):** - `GET /api/docs/tree` (admin), `build_kb_tree` pure builder, `PATCH /api/folders/summary`, migration 0018 (`manually_edited`, head confirmed), generator skip/keep + `kept_manual` stat, RAG tree UI + edit affordance in `sources.js`/`index.html`/`styles.css` - `tests/e2e/test_kb_tree.py`: 8 passed — top level, drill source/folder, edit round-trip, clear, manual-desc-survives-sync, reload fallback, anonymous gate - Integration: tree shape/order/403/empty/indexed-only + PATCH update/create/root/clear/404/403/no-LLM + stat-walk equivalence (in `test_docs_api.py`); 3-field `folder_summaries=` import token preserved **Gates (exact commands):** - `uv run pytest --cov=app --cov-report=term-missing` → **2053 passed**, TOTAL coverage **99%** (>90% ✓) - `uv run ruff check . && uv run pyright` → **All checks passed / 0 errors** - `uv run pytest tests/e2e/test_kb_tree.py -v --no-cov` → **8 passed** in isolation - 30 story/RAG-view E2E suites run **one per process**: all passed, incl. `test_ls_tree_drilldown` (agent `ls` byte-identical ✓), `test_import_documents`, `test_edit_summaries`, `test_admin_auth`, `test_kb_overview` **Completion criteria:** tree view ✓ · edit round-trip + clear ✓ · manual persists/clear resets ✓ · `ls` unchanged ✓ · pytest/coverage/lint ✓ · E2E isolation ✓ · commit — left to harness per protocol (working tree untouched, `git add/commit` not run) **Deviations:** none. **Next pending phase:** none — `todo/` contains only 97 (96 already committed). |
||
|
ducoterra
|
a49be80b8e |
phase: 96_oneshot_resilience
All checks complete. Final report: **Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design) - `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff - `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe - `.env.example` comments updated (chat-turn stream + one-shot summary calls) **Test / lint / coverage results** - `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%) - `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated) - Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files). **Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`. |
||
|
ducoterra
|
bcaef800c5 |
phase: 95_read_truncation_cap
All gates are green. Final verification is complete — no fixes were needed; the phase is fully implemented and passing. **Phase 95 final verification report** - Verified all 3 tasks already implemented (task files in `complete/`): read cap + notice + `ToolResultPiece` (task 01), `tool_result` SSE + live/saved/shared marker (task 02), E2E suite (task 03). - Code checks passed: `BOR_READ_MAX_CHARS` (default 128 000, validator), pinned `READ_TRUNCATION_NOTICE` + `TRUNCATION_MARKER`, byte-identical at/under-cap read, holder `read_truncations` (no `tool_calls` impact), piece order (after `tool`, before next round), `ChatToolResultEvent`, `ToolCall` back-compat fields, `app.js` live/restore handlers, `shared.js`, `.truncated-note` CSS, `.env.example` entry, mock-LLM echo script. - **Tests:** `uv run pytest` → 1966 passed; `uv run pytest --cov=app --cov-report=term-missing` → all green, TOTAL **99%** (>90% gate). - **E2E:** `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` → **3 passed** (frame order + live marker + LLM notice via echo; save→shared fidelity; short-read control). - **Regression (isolated):** `test_agent_document_tools` 4 ✓, `test_chat_history` 5 ✓, `test_share_chat` 4 ✓, `test_big_read_progress` 4 ✓, `test_stop_generation` 3 ✓. - **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings. **Completion criteria:** ① over-cap read → first-cap-chars + marker + pinned notice — ✓ (unit-pinned: at-cap/cap+1/notice tests); ② user marker live/saved/shared — ✓ (E2E + frontend tests); ③ at/under cap byte-identical, no frame — ✓ (unit + control E2E); ④ top-2 `<documents>` retrieval untouched — ✓ (`app/rag/retriever.py` unmodified vs HEAD); ⑤ suite green, >90% coverage, ruff+pyright clean — ✓; ⑥ no completed-phase behavior change — ✓ (all gates green; commit left to harness per pass rules). - No defects found; no changes made this pass. Next pending phase: none in `todo/` (96 is the next free number). |
||
|
ducoterra
|
d4943b4822 |
phase: 94_ls_tree_drilldown
All green. Verification complete. **Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)** - Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal - Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths - Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met - `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %) - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched) - Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed - Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol) **Next pending phase:** `95_read_truncation_cap` |
||
|
ducoterra
|
8c706259e9 |
phase: 89_source_ignore_paths
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed. ## Phase 89 — final verification pass: ALL GREEN **Verified (all 6 task files present in `complete/`):** - `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`) - Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources` - API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403) - Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py` - Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box **Test/lint results:** - `uv run pytest` → 1808 passed - `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%) - `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up) - Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules. **Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green). **Next pending phase:** none — `todo/` holds only this phase. |
||
|
ducoterra
|
f4150421bb |
phase: 84_docs_push_error_sanitization
**Phase 84 — final verification pass: all green, no defects found** - Verified implementation: `app/core/errors.py` (verbatim lift of sync masker), `app/api/sync.py` alias import, docs-push 502 `detail=sanitize_error(str(exc))`, all five `llm.py` error sites sanitized; new/extended test pins in place - Tests: `uv run pytest` → **1714 passed, 0 failed**; targeted pins (new unit ×2 + integration ×1, existing 502 pin) → 13 passed; sync/git-sources regression → 67 passed - Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **99%** (`app/core/errors.py` 100%, `app/rag/llm.py` 100%) — >90% met - E2E isolation: `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → **3 passed** - Lint/types: `uv run ruff check .` → clean; `uv run pyright` → **0 errors** - Criteria: 502 masks `*****@`/never token + row untouched ✅; LLM base-URL masked, credential-free strings byte-identical ✅; `_CREDS_RE` only in `app/core/errors.py` (working-tree grep) ✅; full gate green ✅; `git diff --stat` limited to the 4 app files + 2 modified test files + 3 phase task files (untracked: new module, new unit test, complete/ dir, reports, audit plan) ✅ - Commit/phase move left to the harness per instructions (task files already in `complete/`) - No deviations; nothing to fix - Next pending phase: **85_mobile_menu_gate_overlap** |
||
|
ducoterra
|
894637108c | add configurable llm timeout | ||
|
ducoterra
|
055c0b5d85 |
feat(rag): pass chat history with prior thinking to the LLM
Phase 74 (TODO.md L4): a follow-up question now reaches the model WITH the conversation so far — every prior user/brain turn and the prior thinking blocks on brain turns (preserve-thinking) — while POST /api/chat stays stateless (A10): the client provides the history in the request body and the server stores nothing new. Server (task 01): - ChatRequest.history: optional list[HistoryTurn] (who: user|brain, text, optional thinking) — absent/empty keeps the request byte-identical to pre-phase-74 (the two-message [system, user] request; the kill-switch semantics are pinned in the integration suite). - app.rag.prompts.history_to_messages: pure mapper — walks the turns newest-first against the settings budgets (history_max_turns=40 / history_max_chars=24000, BOR_HISTORY_MAX_TURNS / BOR_HISTORY_MAX_CHARS); a capped turn is dropped WHOLE (never cut mid-answer); the kept window is returned oldest-first; brain turns carry their thinking as reasoning_content (A4) only when non-empty. - Both branches feed it: the deflected path splices it between the system prompt and the current user message (the phase-71 recovery still rebuilds from messages[1:]), the grounded agent receives run_agent(..., history=hist); llm.py's message params widen to list[dict[str, Any]] (string-only messages stay byte-identical on the wire — the SDK passes message dicts through verbatim). - The per-turn log line (PLAN §9) gains history_msgs=N after kb_chars=N. - Pins: tests/unit/test_history.py (mapper: mapping, reasoning gating, both budgets, drop-whole, ordering, empty default), tests/unit/test_config.py (the two settings + env overrides), tests/unit/test_agent.py (the history splice + the default), tests/integration/test_chat_api.py (deflected AND grounded forward the history incl. reasoning_content, no-history byte-identity, 422 pins, the log field). Client (task 02): - runTurn — the single funnel for fresh send / phase-49 retry / phase-53 stale-regen — sends history = the conversation record minus the current question, with thinking only on brain records that streamed one (undefined drops the key from the JSON, the record's convention); the question is never duplicated into the history. Wire proof (task 03): - The mock's echo my history marker (HISTORY_TRIGGER) answers with the deterministic history echo — history: N prior messages; last answer tail: <last 24 chars>; thinking: yes|no — checked BEFORE the DEFLECT_MODE branch (like TABLE_TRIGGER), so it fires on both turn branches whatever the gate says; the module docstring records the user/assistant-only history invariant that keeps every existing (tool-result-classified) marker flow unaffected. - tests/e2e/test_llm_history.py (isolated): a grounded follow-up and a deflected follow-up both receive history: 2 prior messages + thinking: yes + the byte-exact tail of turn 1's answer (derived from the persisted bor.chat.v1 record — the same array the client maps into the body); a cold start receives history: 0 prior messages / last answer tail: none / thinking: no. - Regressions green in isolation: chat_rag, chat_history (phase 50), agent_document_tools, harness_aligned_tools, stop_generation, retry_answer, response_to_docs. |
||
|
ducoterra
|
766702c750 | finally getting accurate answers | ||
|
ducoterra
|
988ff78526 |
fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled methodology (owner directive: stop clearing/re-importing the homelab KB per iteration; measure tool-calling accuracy on a controlled fixture KB, target >90%). Real-model gate verdicts (live, configured chat model 'lite', fixture KB): - Controlled fixture battery (the new methodology's pass condition — contract accuracy >= 90%): PASS, 4 consecutive runs: gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s) gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s) - Locked derived battery (phase-72 task 05, executed >= 90% bar, run unchanged on the same fixture KB): gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s) The teaching works — every bare-path trap self-corrects in exactly one round, zero cap hits, zero repeat loops, 10/10 answered. The locked executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the corrected re-reads (the trap question seeds its target, so the correct combined-form read is refused for redundancy) — a copy-invariant model behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04) and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md sections 5 and 7), not a copy lever. Copy changes this phase owns (unit pins updated to follow): - app/rag/agent.py: ls teaching refusals (path-like scope -> document-path line; unknown source -> no-source line with the source-name parenthetical), read/grep 'did you mean source/path?' teaching (find_path_candidates: exact or suffix path match, catalog order, cap 3), ALREADY_IN_CONTEXT naming the correct action (answer from the text already in the prompt), read tool description front-loaded with the do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is the only remaining refusal class; contract accuracy 92-100% across runs) - app/rag/prompts.py: TOOLS_SECTION states the document-identity contract up front (ls path = source name; read/grep = combined source/path including the source name; do-not-read for <documents> documents placed next to the read teaching; one-call-per-reply and never-repeat rules) - tests: refusal pins (unit + integration), new dedicated E2E suite tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in isolation), regression suites green in isolation (harness_aligned_tools, agent_document_tools, agent_unlimited_tools, search_tool, chat_rag). Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff + pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/ move and both phases' .agent/reports/ (AGENTS.md 8). |
||
|
ducoterra
|
575d6c88d0 | feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery | ||
|
ducoterra
|
801639efcc | feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?) | ||
|
ducoterra
|
0bf96f22e1 |
fix(agent): make read_document robust to combined source/path arguments
The model treated the combined 'source/path' string (as printed in search result lines, read-result headers and refusals) as the document's identity and passed it as 'source' — e.g. source='homelab/active/container_caddy/caddy.md' instead of source='homelab', path='active/container_caddy/caddy.md'. - Rewrite the read_document description with the split rule (source = before the FIRST '/', path = after it) and a worked example; share the source/path parameter descriptions between read_document and search_documents; map search result lines back onto the split. - New _resolve_document: on a lookup miss with a '/' in source, retry at the first slash (source names are directory basenames and can never contain '/'), plus a continuation candidate for a split at a later slash; a self-corrected combined form for an already-in-context document is still rejected as ALREADY_IN_CONTEXT. - A slash-carrying source that matches nothing gets an educational refusal naming the corrected arguments instead of the generic line that repeated the combined form. Verified live against aipi (lite) + the imported homelab KB: A/B on the exact failure scenario (5 runs each, right after a combined-source search result) — old descriptions 5/5 combined, new descriptions 5/5 clean; two live UI turns (Playwright) produced only clean split arguments, including a multi-hop read of install_caddy_deskwork.yaml that landed in done.sources. Full suite: 1376 passed, app coverage 99% (agent.py 100%), ruff + pyright clean, agent/search E2E green in isolation. |
||
|
ducoterra
|
137d5fa1a5 | feat(sources): removing a source deletes its files and index entries behind a confirmation modal | ||
|
ducoterra
|
189aa92182 | fix tool descriptions to prevent incorrect tool calls | ||
|
ducoterra
|
8cf3a827ee | feat(agent): search_documents tool — the model can grep the indexed documents for an exact string | ||
|
ducoterra
|
88293ed02f | feat(rag): retry a failed LLM request before the first token lands — BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY with a live 'retrying' status | ||
|
ducoterra
|
4677d86f49 | feat(sources): real-time file progress for sync and upload — background upload with success toast | ||
|
ducoterra
|
15a16a8fe0 | fix(agent): unambiguous document listing format for LLM parsing | ||
|
ducoterra
|
d94f3d5a52 | feat(import): user-extensible BOR_IMPORT_EXTENSIONS — any well-formed extension, A9 family stays the default | ||
|
ducoterra
|
32b7bfd4b3 | feat(chat): invalidate saved chats on sources sync — versioned stamps, stale marker, Regenerate against the new index | ||
|
ducoterra
|
1a60ecbd8b | feat(chat): stop an in-flight answer — Send becomes Stop, the partial is kept and persisted, the model stream is torn down | ||
|
ducoterra
|
03d26255c6 |
feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan … reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content" — design confirmed in the same conversation): * POST /api/git-sources/upload (admin-only, require_admin): accepts .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap (bounds BOTH the compressed upload and the total extracted bytes — zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink escape and device/FIFO members rejected), and atomically swaps the content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the archive suffix — no missing window, a failed upload never touches the existing folder/row/KB). The git_sources row is upserted by path (kind='local', no duplicates, added_at preserved), the models are checked fail-fast (503 sanitized when down — the folder/row stay committed and the next sync/re-upload retries idempotently), and the source is scanned synchronously in the request (single-source import_sources prune=True + change-gated KB overview), answering 200 with the sync-style counts. One upload at a time (409); the request session is released before the scan so a concurrent TRUNCATE cannot deadlock against it. * app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES, archive_source_name (safe-name derivation), unpack_archive (guarded zip/tar extraction with the extracted-byte cap, no partial state), swap_in (atomic replace with restore-on-failure) — fully unit-tested. * app/config.py + .env.example: BOR_UPLOAD_DIR (default ~/bor-sources/uploads, deliberately separate from the git checkouts) and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at startup on <= 0). * python-multipart added to the dependencies — FastAPI's required multipart parser (an A2 implementation detail, phase locked decision). * The Sources page: the phase-38 "Add a local directory" form is removed; #archive-upload-form takes its place (labeled file input, "Upload & scan" button, the §7.4 never-stale lifecycle, inline role=alert error, role=status count line); hint + table caption updated. The POST /api/git-sources kind=local API contract is UNCHANGED — a plain directory is still registrable via the API, and existing Local rows list/remove/sync exactly as before. * The phase-38 story E2E (test_local_directory_sources.py) is rewritten API-driven — the form it drove is gone; its acceptance stands. * The story E2E (test_archive_upload_sources.py): the swap, upload→scan→list (the deterministic "Uploading…" in-flight state, the Local row, /api/docs + the RAG catalog), same-filename re-upload (in-place replace, prune, no duplicate row, v2-only folder), the 422 inline error + recovery (the form is not wedged), and the anonymous gate + 403. * README: the archive-upload section (formats, naming rule, in-place replace, both new settings), the local-directory form removal noted, config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB. Gates: unit+integration green, app/ coverage 99%, the story E2E green in isolation, the regression suites (git sources admin, local directory sources, sync button, import documents, nav rename, smoke, shared header) green in isolation, ruff + pyright clean. Note: per this phase's file-level staging, frontend/assets/styles.css also carries the small same-day in-flight owner rework already in the working tree (the .sign-in-mobile companion rule for the phase-48 mobile sign-in copy); the phase-49 change is the upload form's block. |
||
|
ducoterra
|
5d679f5184 |
feat(import): index quadlet unit files and jinja templates (A9 revision)
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.
- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
import_extensions CSV gain the ten names (the original seven first);
the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
(HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
default CSV + dotted import_extension_set include the ten, the
validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
new suffix (parametrized), the .container fixture chunks >=2 under
the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
tree indexes exactly the ten new files (unknown/hidden/excluded
filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
over a temp tree with .container/.volume/.j2 -> documents + chunks
rows with stem titles; delta re-import updates only the changed .j2
doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
isolation): GET /api/docs (admin session) lists the four new-format
docs with non-zero chunk counts and stem titles; the Sources table
renders a row + .doc-link per file; the phase-26 modal shows the
.container TOML ([Container] section + sentinel) with stem title and
the container format badge; a RESE-JINJA-SENTINEL-33dd question
FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
FTS hits) — the bubble is not .is-deflected and a source chip names
templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
the revision note under the anchors table, and the §5 chunking-policy
+ §11 workflow lines. The only PLAN edit this phase.
Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.
Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
|
||
|
ducoterra
|
b855d0aef9 |
feat(rag): unbounded agent tool calls behind a round cap (owner revision)
Phase 45 (owner permission 2026-08-27, TODO.md L8: "allow the LLM
to make as many tool calls as it wants"): the phase-37 per-turn tool
budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each)
and their exhaustion refusals are removed — a grounded turn now offers
list_documents / read_document for the whole turn (re-lists included),
bounded only by the round cap:
- app/config.py: agent_max_rounds (BOR_AGENT_MAX_ROUNDS, default 10,
negative rejected) replaces agent_list_calls / agent_read_calls;
.env.example + README document the single knob; app/rag/prompts.py
docstrings follow.
- app/rag/agent.py: the loop runs tools until the model answers or
rounds >= max_rounds, at which point it forces one final no-tools
answer (the cap is the only forced exit); 0 = no tools — exactly one
tools=None request, byte-identical to the pre-phase-37 path (the
kill switch). Rejected calls (unknown tool / missing args /
already-in-context / unknown path) still consume a round, so
pathological rejected-call streams are bounded by the cap. The
per-call log line is now tool/args/round=N/M; the per-turn
tool_calls=N field and the tool SSE event are unchanged.
- tests/e2e/mock_llm.py: MULTI_READ_TRIGGER ("read two documents") —
the deterministic list -> read #1 -> read #2 -> forced-answer flow
(byte-stable "I read <sp1> and <sp2>." line), classified by the
count of tool-role read results; the phase-37 single-read flow stays
byte-identical (unit-pinned in tests/unit/test_mock_tool_flow.py).
- tests/e2e/test_agent_unlimited_tools.py (new, story suite,
mock-only): three tool frames/lines in order (one list, two reads —
the second read is what the old read budget refused) + the
both-named non-deflected answer; done.sources + chips = retrieval
doc + both reads, deduped; no budget refusal rendered; the
single-read marker flow regression (exactly one read, single tool
pair).
- .agent/PLAN.md: the phase-45 SSE revision note (owner-locked, R2) —
the only PLAN edit this phase; the phase-37 note's budget clause is
marked removed.
Unit/integration rewrites (test_agent.py round-cap matrix incl. the
kill switch and rejected-call spam, test_config.py, test_chat_api.py
agent_max_rounds=0 fixtures) landed with the server core so every gate
stays green.
uv run pytest: 756 passed, app/ coverage 99%; ruff + pyright clean;
story E2E 4/4 in isolation (ran twice); regression E2E suites
(agent_document_tools unmodified, chat_rag, smoke) green in isolation.
Also records the 45_agent_unlimited_tools todo/ -> complete/ task-file
moves (00/01/02 pending in the working tree, task 03 moves on success).
|
||
|
ducoterra
|
6cf1df9bf2 |
feat(sync): fail fast with a modal when a model is unavailable
TODO.md L4: with a dead model endpoint the sync discovered it only mid-import, after slow clones — and a tooltip on the button is not a readable error. - app/rag/llm.py: ModelUnavailableError + check_models(llm) — a tiny pre-sync probe (one short embedding + one 1-token-scale completion) that fails naming the unavailable model (embed first, then the summary model); the sync sanitizer still masks credentials. - app/api/sync.py: the probe is step 1 of _run_sync — before source resolution and before any clone_or_pull; a model failure is just another 'failed' state (no new endpoint, A10/A12 untouched). - frontend/assets/header.js: applySyncFailure now also opens the module-owned error modal (every page carrying #sync-btn, zero page-markup changes): lazily built backdrop + role=alertdialog panel, error text via textContent, close via button / Esc / backdrop, focus in-and-out to #sync-btn (with a body→#sync-btn fallback — the run's disabled button drops focus to <body>). - frontend/assets/styles.css: the modal on the phase-08 error palette (z-index above the header, .is-open open/close, reduced-motion stilling, 44px close target). - Tests: probe unit tests (both up / embed down / summary down / custom model names), sync integration (fail-fast before any clone, probe-before-effective_sources ordering, credential masking, healthy regression), the phase-41 source pins, and the story E2E (two module apps on distinct ports — dead endpoint on a closed loopback port vs session mock: ≤10 s fail-fast + modal contract, all three dismissal paths with focus out to #sync-btn, button title/.is-error + Sources banner untouched, healthy phase-32 lifecycle regression to 'Synced HH:MM'). E2E (isolation): test_sync_model_down.py 4/4, test_sync_button.py 3/3, test_git_sources_admin.py 6/6, test_local_directory_sources.py 3/3; unit+integration 721 passed, app/ coverage 99%; ruff + pyright clean. |
||
|
ducoterra
|
94d7228510 |
feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
|
||
|
ducoterra
|
15c1272828 |
feat(rag): agent document tools — list/read tools with env-tuned budgets, SSE tool events + "calling tool" UI
Grounded chat turns now run the agent loop (app/rag/agent.py) instead
of a bare chat_stream: while the per-turn budgets last
(BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each) the model
gets list_documents (the indexed catalog, /api/docs order) and
read_document (full text, never truncated — A7-revised contract); once
both budgets are spent the tools key is dropped from the request and
the model must answer. Rejected calls (unknown tool, unknown/missing
path, document already in context, spent budget) consume no budget.
Budgets 0/0 make exactly one tools=None request — byte-identical to
the pre-phase path (budgets-as-kill-switch). Deflected turns keep the
direct chat_stream (A8 unchanged; the LOW prompt never carries the
<tools> section).
SSE contract gains {"type":"tool","name":...,"argument":
"source/path"|null} frames ahead of the answer deltas (PLAN §4
extension, owner permission 2026-08-26); done.sources, query_log.sources
and the per-turn log line (gains tool_calls=N) report the retrieval
docs + read docs, deduped. The UI shows a "calling tool"
button/label state and one visible .tool-call line per call above the
answer; the lines persist with the chat record and re-render on
reload. chat_stream passes tools through and accumulates streaming
tool_calls deltas into ToolCallPiece (tools=None stays byte-identical).
E2E: deterministic mock tool flow ("use your tools" + <tools> marker:
list -> read first catalog line -> quoted answer) plus the story suite
(marker flow, reload re-render, plain/deflected no-tool regressions).
Docs: .env.example + README (the two tools, the budgets, the SSE tool
frame, the "calling tool" UI state).
probe: turbo tool_calls=supported 2026-08-26 (uv run python -m
scripts.llm_probe --tools — non-streaming + streaming
finish_reason=tool_calls, indexed delta.tool_calls partials)
|
||
|
ducoterra
|
1925bb66a8 | feat(sources): admin page to add and remove git sources (TODO.md L4) | ||
|
ducoterra
|
0654b304e1 | feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts | ||
|
ducoterra
|
572a4190a6 | feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc |