phase: 94_ls_tree_drilldown
All green. Verification complete. **Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)** - Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal - Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths - Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met - `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %) - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched) - Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed - Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol) **Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
@@ -0,0 +1,14 @@
|
|||||||
|
All green. Verification complete.
|
||||||
|
|
||||||
|
**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**
|
||||||
|
|
||||||
|
- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
|
||||||
|
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
|
||||||
|
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
|
||||||
|
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
|
||||||
|
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
|
||||||
|
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
|
||||||
|
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)
|
||||||
|
|
||||||
|
**Next pending phase:** `95_read_truncation_cap`
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 14%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 110 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 133 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 124 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 306 1 99%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 110 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 269 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3647 13 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+17
@@ -0,0 +1,17 @@
|
|||||||
|
Task 01 is complete and fully verified.
|
||||||
|
|
||||||
|
**Implemented** (phase 94, task 01 — folder summary storage + generator)
|
||||||
|
- `alembic/versions/0017_folder_summaries.py`: `folder_summaries` table, PK `(source, folder_path)` mirroring `documents` String(120)/String(1000), TEXT `summary` NOT NULL, TIMESTAMPTZ `updated_at` (now() default); tested downgrade
|
||||||
|
- `app/models.py`: `FolderSummary` model (`folder_path = ""` = source root; ≥2-doc policy documented)
|
||||||
|
- `app/rag/folder_summaries.py`: `FOLDER_SUMMARY_MODE` marker + 1–3-sentence locked instruction, `folder_of`, `group_by_folder` (recursive subtree = exactly the `ls` count rule — I fixed a gap: the `path == folder` arm now lands a file-sharing-a-directory-name in its folder's group), capped prompt builder (8 000 chars, `BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS` + `TRUNCATION_MARKER`), `summarize_folder` (empty→`LLMError`), `generate_folder_summaries` (skip no-op, per-folder fail-soft, ≥2 rule, stale prune, flush-only caller-commits, stats dict + log line)
|
||||||
|
- `tests/e2e/mock_llm.py`: canned `Fixture folder summary for <folder>.` branch, checked **before** the `SUMMARY_MODE` branch (marker-substring shadowing — real bug, pinned)
|
||||||
|
- Tests: 31 unit + 4 migration integration; also repaired dev DB (found at 0006 → upgraded to 0017 head)
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **1903 passed, TOTAL 99%** (`folder_summaries.py` 100%, `models.py` 100%)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
- E2E in isolation, all PASS: `test_harness_aligned_tools`, `test_agent_document_tools`, `test_agent_unlimited_tools`, `test_search_tool`, `test_grep_regex_teaching`, `test_response_to_docs`, `test_kb_overview`, `test_document_summaries`
|
||||||
|
|
||||||
|
**Notable** — a previous (a1) attempt's work was in the tree; I verified rather than trusted it, fixed the `group_by_folder` concept gap, and added the mock-branch + edge-case pins.
|
||||||
|
|
||||||
|
**Next pending task** — `.agents/phases/todo/94_ls_tree_drilldown/02_sync_wiring.md`
|
||||||
+94
@@ -0,0 +1,94 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
............................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 133 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 124 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 108 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 269 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3552 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
(no final assistant message — see the .err log)
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
............................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 133 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 124 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 108 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 269 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3552 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
All green. Everything for task 02 is implemented, tested, and verified.
|
||||||
|
|
||||||
|
**Report — task 02 (sync wiring) completed**
|
||||||
|
|
||||||
|
- `scripts/import_docs.py`: `_run()` regenerates `folder_summaries` under the overview's change gate (added+updated>0) or the table-empty first-run trigger; same event loop/LLMClient; generator flushes, the run's own short-lived session commits (phase-53 convention); stats on the summary line as `folder_summaries=<gen>/<fail>/<prune>` or `skipped`; docstrings updated
|
||||||
|
- `app/api/sync.py`: step 5 calls the same generator + gate (change or empty table); per-folder fail-soft (a `lite` outage never flips the run and never blocks the bump); stats log-only — `/api/sync/status` detail shape untouched
|
||||||
|
- `app/rag/folder_summaries.py`: added `folder_summary_table_empty()` gate probe (LIMIT 1)
|
||||||
|
- New `tests/integration/test_sync_folder_summaries.py` (10 tests: changed/unchanged/prune/fail-soft/`--limit`/empty-table on both paths); updated overview/git/sync-API/sync-button suites (pinned line endings + layer stubs per the line-extension rule); +1 unit test for the gate probe
|
||||||
|
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **1914 passed**, app/ coverage **99%** (>90%)
|
||||||
|
- `uv run pytest tests/integration/test_sync_folder_summaries.py -v --no-cov` → 10 passed
|
||||||
|
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
|
||||||
|
|
||||||
|
Decisions: line token format `G/F/P|skipped`; `--limit` skip enforced at the caller (early return) with the generator's `skip` flag as tested backstop; API folder gate = change ∨ empty table, overview gate unchanged. Next pending: `03_ls_tree.md`.
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
.......................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 110 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 133 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 124 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 222 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 110 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 269 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3563 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
All gates green. Task complete — final report:
|
||||||
|
|
||||||
|
**Phase 94 · task 03 (drill-down `ls`) — DONE, no commit**
|
||||||
|
|
||||||
|
- `app/rag/agent.py`: `ls` branch rewritten to the 3-level tree — `ls_top`/`ls_folder` (+ pure `group_folder_listing`, `render_ls_top`/`render_folder_listing`); new `NOT_A_FOLDER` (parent's subfolders), `LS_PATH_NOT_A_SOURCE` deleted; 50-file cap + grep-pointer note; dead `list_catalog` removed; `AGENT_TOOLS` description pinned.
|
||||||
|
- `app/rag/prompts.py`: TOOLS_SECTION drill-down teaching (read/grep clauses byte-identical).
|
||||||
|
- `app/api/chat.py` + `scripts/agent_realmodel_check.py`: comment/script repointed off the deleted accessor.
|
||||||
|
- Tests: unit (agent/prompts/mock-flow) + integration (real-DB tree, cap, refusals, ordering) rewritten; 5 E2E suites updated for the drill frame (git_sources TRUNCATE + commit-between-inserts seed ordering — UUID tie-break is random, `added_at` is per-transaction).
|
||||||
|
- Gates: `pytest` **1939 passed**; `app/` coverage **99%** (gate >90%); ruff clean; pyright 0 errors; E2E in isolation — document-tools 4✓, unlimited 4✓, harness-aligned 3✓, path-teaching 2✓, big-read 4✓, search 3✓, grep-teaching 2✓.
|
||||||
|
|
||||||
|
Next pending: `04_e2e_drilldown.md` (not started).
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 14%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 110 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 133 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 124 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 306 1 99%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 110 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 269 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3647 13 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
All green. Task complete.
|
||||||
|
|
||||||
|
## Report — Phase 94, Task 04 (E2E drill-down story suite)
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `tests/e2e/test_ls_tree_drilldown.py` (new, isolated, own module app port 8136): two local temp sources (`alpha` 56 docs incl. a 51-file `wide/` folder, `beta` 2 docs) registered via the authenticated API + real in-process `POST /api/sync`; pins the 6 stored `folder_summaries` rows byte-exactly (mock's canned `FOLDER_SUMMARY_MODE` text)
|
||||||
|
- 3 tests: 4-turn scripted drill `ls()` → `ls(alpha)` → `ls(alpha/two)` → `read(alpha/two/two-a.md)` (tool lines, wire `tool` frames, exact tree levels incl. summaries + file-line format, grounded citations); 51-file cap (exactly 50 lines + `…and 1 more…` note, 51st absent); NOT-A-FOLDER teaching + scripted recovery
|
||||||
|
- `tests/e2e/mock_llm.py`: new `DRILL_TRIGGER` scripted flow (question carries its own call: `drill down the tree: ls [target]` / `read source/path`) — tool call → echo the listing verbatim into the answer (house lens); recovery branch keys on receiving the NOT-A-FOLDER line; ALREADY_IN_CONTEXT branch answers from the `<documents>` block
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov` → 3 passed (in isolation, re-run twice)
|
||||||
|
- Regression gate in isolation: `test_agent_document_tools`, `test_agent_unlimited_tools`, `test_harness_aligned_tools`, `test_search_tool`, `test_grep_regex_teaching`, `test_sync_button`, `test_local_directory_sources`, `test_response_to_docs` → all exit 0, 0 failures
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, app coverage 99% (>90%)
|
||||||
|
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors
|
||||||
|
|
||||||
|
**Notable decision**: the read target (its path is named in the question) deterministically ranks top-2, so the agent's phase-72 dedupe returns `ALREADY_IN_CONTEXT`; the mock then answers from the `<documents>` prompt with the same citation shape (prefixed `Already in context:`), which the suite pins — the document citation contract holds and the dedupe notice's visibility is asserted.
|
||||||
|
|
||||||
|
**Next pending task**: `.agents/phases/todo/94_ls_tree_drilldown/05_realmodel_battery.md`
|
||||||
+94
@@ -0,0 +1,94 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 14%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 110 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 133 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 124 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 306 1 99%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 110 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 269 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3647 13 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
Task 05 complete — all green.
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- Fixture dump refreshed: `load_test_kb` now generates sync-time folder summaries against live `lite` (4 rows: 2 source roots + `homelab/containers` + `homelab/networking`) and carries `folder_summaries` in TRUNCATE/INSERT/round-trip verify; `restore_test_kb` reports them — restore 0.03 s hot
|
||||||
|
- Fixed stale gate-side mirror: `classify_call` now scores `ls(source/folder)` drill-downs as contract-correct (mirrors task-03's resolution rules; previously the correct new behavior would have been flagged)
|
||||||
|
- Ran the 10-turn battery against the live endpoint, both models: configured `turbo` (verdict) and `lite` (same-model check vs the phase-70/72 standing, `.env` switched then restored byte-identical)
|
||||||
|
- Recorded the dated phase-94 section in `TOOL_CALLING_TESTING.md` (per-turn lines, verdicts, baseline table, conclusion) + moved Q1/2/3/10 expected-behavior notes with the owner-permitted surface (questions untouched) + §2/§5/§8 notes
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- turbo: `PASS … calls 18/19 executed (95%) contract 19/19 (100%) (wall 98.7s)` — vs 112.8–113.7 s baseline: −12.5…−13.2 %, within band
|
||||||
|
- lite: `PASS … contract 18/18 (100%) (wall 43.6s)` — vs 40.4–40.5 s standing: +7.7 %, within band; no copy-lever iteration needed
|
||||||
|
- `uv run pytest --cov=app` → rc=0 (1939 passed, 2 skipped), TOTAL 99 %; `uv run ruff check .` clean; `uv run pyright` 0 errors
|
||||||
|
|
||||||
|
**Decisions**
|
||||||
|
- Verdict run uses the configured model (`turbo`); `lite` run added for the same-model comparison the task's "phase 70/72 numbers" reference implies — both recorded
|
||||||
|
- Flagged for the owner (not a gate miss): `lite` answered the two "list everything" turns from the top level alone; lever noted in §9 if file-level depth is wanted
|
||||||
|
|
||||||
|
**Next pending task:** `95_read_truncation_cap/01_read_cap.md`
|
||||||
+94
@@ -0,0 +1,94 @@
|
|||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 14%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 178 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 94 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 229 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 29 0 100%
|
||||||
|
app/api/sync.py 110 0 100%
|
||||||
|
app/api/tokens.py 28 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 133 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 33 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 66 0 100%
|
||||||
|
app/models.py 124 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 306 1 99%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/folder_summaries.py 110 0 100%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 190 3 98%
|
||||||
|
app/rag/llm.py 217 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 88 0 100%
|
||||||
|
app/rag/retriever.py 150 3 98%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 269 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 3647 13 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
@@ -36,6 +36,7 @@ BOR_STEERING_MAX_CHARS=8000 # char budget for the <tuning> (steering notes
|
|||||||
BOR_SUMMARY_MAX_CHARS=12000 # cap on document content sent to the lite summary model (phase 30)
|
BOR_SUMMARY_MAX_CHARS=12000 # cap on document content sent to the lite summary model (phase 30)
|
||||||
BOR_KB_OVERVIEW_MAX_CHARS=4000 # char budget for the <knowledge_base> prompt section (phase 31)
|
BOR_KB_OVERVIEW_MAX_CHARS=4000 # char budget for the <knowledge_base> prompt section (phase 31)
|
||||||
BOR_OVERVIEW_INPUT_MAX_CHARS=40000 # cap on the document list sent to the lite model for the KB outline (phase 31)
|
BOR_OVERVIEW_INPUT_MAX_CHARS=40000 # cap on the document list sent to the lite model for the KB outline (phase 31)
|
||||||
|
BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS=8000 # cap on a folder's document list sent to the lite model (phase 94)
|
||||||
BOR_CHUNK_TARGET_CHARS=2000
|
BOR_CHUNK_TARGET_CHARS=2000
|
||||||
BOR_CHUNK_OVERLAP_CHARS=200
|
BOR_CHUNK_OVERLAP_CHARS=200
|
||||||
BOR_EMBED_BATCH_SIZE=16
|
BOR_EMBED_BATCH_SIZE=16
|
||||||
|
|||||||
+181
-13
@@ -127,14 +127,21 @@ state is snapshotted to **`tests/fixtures/test_kb.dump.sql`** — a
|
|||||||
data-only SQL script (TRUNCATE + one multi-row INSERT per app table:
|
data-only SQL script (TRUNCATE + one multi-row INSERT per app table:
|
||||||
documents, chunks + embeddings, the `git_sources` local rows that make
|
documents, chunks + embeddings, the `git_sources` local rows that make
|
||||||
the source registry self-contained, the static KB overview, the
|
the source registry self-contained, the static KB overview, the
|
||||||
sources version). Restoring it puts the whole known state back in one
|
sources version, and — since the 2026-09-11 phase-94 refresh — the
|
||||||
transaction; the generated `chunks.tsv` column is recomputed by
|
sync-time `folder_summaries` rows the drill-down `ls` shows). The
|
||||||
Postgres. The dump is verified by round-trip at build time (restore +
|
build generates those rows against the live `lite` endpoint through
|
||||||
per-table checksum compare — a serialization bug fails the build).
|
the real sync-time generator (4 rows for this KB: the two source
|
||||||
|
roots + `homelab/containers` + `homelab/networking` — the ≥ 2-document
|
||||||
|
folders; single-document folders carry no row by design). Restoring it
|
||||||
|
puts the whole known state back in one transaction; the generated
|
||||||
|
`chunks.tsv` column is recomputed by Postgres. The dump is verified by
|
||||||
|
round-trip at build time (restore + per-table checksum compare — a
|
||||||
|
serialization bug fails the build).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Rebuild the KB + dump — only when the fixture documents, the
|
# Rebuild the KB + dump — only when the fixture documents, the
|
||||||
# chunker, or the embedding model change. NOT part of the loop.
|
# chunker, the embedding model, or the folder-summary prompt change
|
||||||
|
# (its `lite` output is baked in — phase 94). NOT part of the loop.
|
||||||
uv run python -m scripts.load_test_kb
|
uv run python -m scripts.load_test_kb
|
||||||
# restores the fixture KB standalone (what --restore runs inline)
|
# restores the fixture KB standalone (what --restore runs inline)
|
||||||
uv run python -m scripts.restore_test_kb
|
uv run python -m scripts.restore_test_kb
|
||||||
@@ -170,22 +177,31 @@ of it.
|
|||||||
|
|
||||||
| # | question | tests | expected ideal |
|
| # | question | tests | expected ideal |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 1 | List the files in this directory. | the phase-72 incident; full listing needs `ls` (8 docs, 2 in seed) | `ls()` |
|
| 1 | List the files in this directory. | the phase-72 incident; the top level lists the sources — a full file listing drills them (phase 94) | `ls()` → drill `ls('deployments')` + `ls('homelab')` (+ leaf folders) |
|
||||||
| 2 | List the documents you have in the homelab source. | scoped `ls` by the correct source name | `ls(path='homelab')` |
|
| 2 | List the documents you have in the homelab source. | scoped `ls` by the correct source name; the result is the source's folders (+ counts/summaries) — file lines come from the leaf drills (phase 94) | `ls(path='homelab')` (+ leaf drills) |
|
||||||
| 3 | List every document you have indexed. | no-arg listing | `ls()` |
|
| 3 | List every document you have indexed. | the "list everything" expectation is now the whole tree — no single call lists every file (phase 94) | `ls()` → drill both sources and every folder |
|
||||||
| 4 | Open the document homelab/networking/vela-bridges.md … | `read`, combined form | `read('homelab/networking/vela-bridges.md')` |
|
| 4 | Open the document homelab/networking/vela-bridges.md … | `read`, combined form | `read('homelab/networking/vela-bridges.md')` |
|
||||||
| 5 | Read deployments/quadlet/mimir-service.md and summarize it. | `read`, unseeded target | `read(…)` (or `ls` first, then `read`) |
|
| 5 | Read deployments/quadlet/mimir-service.md and summarize it. | `read`, unseeded target | `read(…)` (or `ls` first, then `read`) |
|
||||||
| 6 | Open the document homelab/networking/meridian-notes.md … | `read`, unseeded target | `read(…)` |
|
| 6 | Open the document homelab/networking/meridian-notes.md … | `read`, unseeded target | `read(…)` |
|
||||||
| 7 | Find the exact string "rbm-8842" in your documents … | `grep`, pattern only | `grep(pattern='rbm-8842')` — the match line alone answers it |
|
| 7 | Find the exact string "rbm-8842" in your documents … | `grep`, pattern only | `grep(pattern='rbm-8842')` — the match line alone answers it |
|
||||||
| 8 | Which document has the title "Lab Ansible Inventory"? Summarize it. | title lookup; target IS seeded | answer from context (or `ls`) |
|
| 8 | Which document has the title "Lab Ansible Inventory"? Summarize it. | title lookup; target IS seeded | answer from context (or `ls`) |
|
||||||
| 9 | What do you know about the qwen 3.8 llama.cpp setup? … | topic lookup; target IS seeded | answer from context |
|
| 9 | What do you know about the qwen 3.8 llama.cpp setup? … | topic lookup; target IS seeded | answer from context |
|
||||||
| 10 | List the files in the deployments directory. | source name phrased as a directory | `ls(path='deployments')` |
|
| 10 | List the files in the deployments directory. | source name phrased as a directory; the result is the source's folders — file lines come from the leaf drills (phase 94) | `ls(path='deployments')` (+ leaf drills) |
|
||||||
|
|
||||||
Questions 4–6 name the **full combined identity** (no bare-path trap —
|
Questions 4–6 name the **full combined identity** (no bare-path trap —
|
||||||
that is the job of the locked derived battery, §6). Questions 7–9 name
|
that is the job of the locked derived battery, §6). Questions 7–9 name
|
||||||
content, so their target document is seeded; the correct behavior
|
content, so their target document is seeded; the correct behavior
|
||||||
there is to **not** re-read what is already in the prompt.
|
there is to **not** re-read what is already in the prompt.
|
||||||
|
|
||||||
|
Phase-94 note: the listing questions' (1, 2, 3, 10) correct behavior
|
||||||
|
**moved with the owner-permitted `ls` surface change** (2026-09-10,
|
||||||
|
`TODO.md` L4 — `00_phase.md` of `94_ls_tree_drilldown`): a full file
|
||||||
|
listing is now a drill sequence (`ls()` → source → folder; one level
|
||||||
|
per call, the old whole-KB flood is structurally gone), and a folder
|
||||||
|
line's count + sync-time summary is the tree's picture of that
|
||||||
|
subtree. The questions themselves are unchanged (the methodology's
|
||||||
|
locked battery) — only the expected-behavior definition moved.
|
||||||
|
|
||||||
### The four pass conditions
|
### The four pass conditions
|
||||||
|
|
||||||
1. all 10 turns answer (no `LLMError`/`MalformedReplyError`);
|
1. all 10 turns answer (no `LLMError`/`MalformedReplyError`);
|
||||||
@@ -205,10 +221,11 @@ gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (6
|
|||||||
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) (wall 46.8s)
|
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) (wall 46.8s)
|
||||||
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 9/12 executed (75%) contract 11/12 (92%) 2026-09-06 (wall 40.4s)
|
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 9/12 executed (75%) contract 11/12 (92%) 2026-09-06 (wall 40.4s)
|
||||||
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/14 executed (64%) contract 13/14 (93%) 2026-09-06 (wall 40.5s)
|
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/14 executed (64%) contract 13/14 (93%) 2026-09-06 (wall 40.5s)
|
||||||
|
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 13/18 executed (72%) contract 18/18 (100%) 2026-09-11 (wall 43.6s) [phase-94 ls tree]
|
||||||
```
|
```
|
||||||
|
|
||||||
Contract accuracy ≥ 90 %: **met** (100 / 92 / 100 / 93 / 92 / 93). The executed
|
Contract accuracy ≥ 90 %: **met** (100 / 92 / 100 / 93 / 92 / 93 / 100). The
|
||||||
ratio sits at 58–73 % for the reason documented in §5 — an app
|
executed ratio sits at 58–73 % for the reason documented in §5 — an app
|
||||||
semantics choice, not a model defect, and the open design question in
|
semantics choice, not a model defect, and the open design question in
|
||||||
§7.
|
§7.
|
||||||
|
|
||||||
@@ -222,6 +239,7 @@ gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (10
|
|||||||
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-06 (wall 113.7s)
|
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 9/9 executed (100%) contract 9/9 (100%) 2026-09-06 (wall 113.7s)
|
||||||
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-06 (wall 112.8s)
|
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 7/7 executed (100%) contract 7/7 (100%) 2026-09-06 (wall 112.8s)
|
||||||
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-06 (wall 97.2s) [derived battery — MISS: 5/10 tool-turn floor]
|
gate: turbo FAIL turns=10 answered=10 caps=0 tool-turns=5 calls 5/5 executed (100%) contract 5/5 (100%) 2026-09-06 (wall 97.2s) [derived battery — MISS: 5/10 tool-turn floor]
|
||||||
|
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 18/19 executed (95%) contract 19/19 (100%) 2026-09-11 (wall 98.7s) [phase-94 ls tree]
|
||||||
```
|
```
|
||||||
|
|
||||||
Reads: the re-read habit is model-specific. `lite` re-reads a seeded
|
Reads: the re-read habit is model-specific. `lite` re-reads a seeded
|
||||||
@@ -304,7 +322,12 @@ The verdict carries both:
|
|||||||
combined `source/path` belongs, a nonexistent document identity, a
|
combined `source/path` belongs, a nonexistent document identity, a
|
||||||
source name where a document belongs (`ls(path='.')`,
|
source name where a document belongs (`ls(path='.')`,
|
||||||
`ls(path='/')`, `grep(path='homelab')` — the entire phase-72
|
`ls(path='/')`, `grep(path='homelab')` — the entire phase-72
|
||||||
incident class).
|
incident class). Phase 94 moved the `ls` contract (owner-permitted
|
||||||
|
surface revision): `ls(path)` is ALSO contract-correct for an
|
||||||
|
EXISTING `source/folder` — the drill-down; the violation is now an
|
||||||
|
unknown first segment, a bare folder name (no source prefix), or a
|
||||||
|
folder matching no indexed prefix. The gate-side mirror
|
||||||
|
(`classify_call` in `scripts/agent_realmodel_check.py`) follows.
|
||||||
- **executed/emitted** (the phase-72 locked metric) = calls the app
|
- **executed/emitted** (the phase-72 locked metric) = calls the app
|
||||||
actually executed ÷ emitted. Every refusal class counts against it
|
actually executed ÷ emitted. Every refusal class counts against it
|
||||||
— **including `ALREADY_IN_CONTEXT`**, the app's dedupe refusal when
|
— **including `ALREADY_IN_CONTEXT`**, the app's dedupe refusal when
|
||||||
@@ -394,7 +417,8 @@ podman compose up -d db
|
|||||||
cp .env.example .env # once; LLM endpoint + DB URL
|
cp .env.example .env # once; LLM endpoint + DB URL
|
||||||
uv run alembic upgrade head
|
uv run alembic upgrade head
|
||||||
|
|
||||||
# 1. Build the controlled KB + dump (one-off, ~2 s — real embeddings)
|
# 1. Build the controlled KB + dump (one-off, ~7 s — real embeddings +
|
||||||
|
# 4 live-lite folder-summary calls since phase 94)
|
||||||
uv run python -m scripts.load_test_kb
|
uv run python -m scripts.load_test_kb
|
||||||
# → prints the retrieval report (all 10 must be grounded) and
|
# → prints the retrieval report (all 10 must be grounded) and
|
||||||
# verifies the dump by round-trip.
|
# verifies the dump by round-trip.
|
||||||
@@ -421,3 +445,147 @@ Diagnosing a bad run: every call is logged by `run_agent`
|
|||||||
refusal templates in `app/rag/agent.py` to see which teaching line the
|
refusal templates in `app/rag/agent.py` to see which teaching line the
|
||||||
model hit, and which refusal class (contract violation vs. in-context
|
model hit, and which refusal class (contract violation vs. in-context
|
||||||
dedupe) the rejection was.
|
dedupe) the rejection was.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Phase 94 — the `ls` drill-down tree (2026-09-11)
|
||||||
|
|
||||||
|
Phase 94 rebuilt `ls` as a drill-down tree (owner-permitted
|
||||||
|
tool-surface revision, `TODO.md` L4 — the `ls` result format and
|
||||||
|
`path` semantics changed; the tool name and the `read`/`grep`
|
||||||
|
contract are untouched): `ls()` lists the synced sources with a
|
||||||
|
per-source sync-time summary, `ls(source)` lists its folders (each
|
||||||
|
with its summary) + its own files, `ls(source/folder)` drills one
|
||||||
|
level deeper; file lines are capped at 50 with a grep pointer. This
|
||||||
|
section is the controlled battery's verdict on that architecture —
|
||||||
|
the TODO's gate: "ensure accuracy and performance aren't too badly
|
||||||
|
impacted", read per the phase-94 task file as **accuracy at or above
|
||||||
|
the recorded baseline** (no contract/verdict regression on the
|
||||||
|
10-question battery) and **wall time within ~20% of the baseline
|
||||||
|
total**.
|
||||||
|
|
||||||
|
**Fixture dump refresh (a precondition of this run).** The 2026-09-04
|
||||||
|
dump predates migration 0017 — it had no `folder_summaries` rows, so
|
||||||
|
the drill-down's summaries would have been absent from the controlled
|
||||||
|
KB. It was rebuilt on 2026-09-11 with
|
||||||
|
`uv run python -m scripts.load_test_kb`: the same 8 hand-written
|
||||||
|
documents through the real pipeline (real embeddings), the static KB
|
||||||
|
overview + sources version re-stored, and the sync-time folder
|
||||||
|
summaries generated against the live `lite` endpoint (4 rows: the two
|
||||||
|
source roots + `homelab/containers` + `homelab/networking` — the
|
||||||
|
≥ 2-document folders). The dump now carries the `folder_summaries`
|
||||||
|
table (TRUNCATE + INSERT + round-trip checksum verification — the
|
||||||
|
house rule); `--restore` measured **0.03 s** hot.
|
||||||
|
|
||||||
|
**The classifier mirror followed the surface.** `classify_call`
|
||||||
|
(`scripts/agent_realmodel_check.py`) is documented as the gate-side
|
||||||
|
mirror of `app.rag.agent._execute_tool`'s resolution rules — task 03
|
||||||
|
changed the `ls` resolution (a `source/folder` path is now valid) but
|
||||||
|
the mirror still flagged every folder drill-down as a contract
|
||||||
|
violation. Fixed in this phase: `ls(path)` is contract-correct for
|
||||||
|
`""`, a registered source, or an EXISTING `source/folder`; without
|
||||||
|
the fix the battery would have scored the *correct* new behavior as
|
||||||
|
inaccurate. (No battery question changed; the §3 expected-behavior
|
||||||
|
notes moved with the owner-permitted surface.)
|
||||||
|
|
||||||
|
**The runs.** Against the live aipi endpoint, the refreshed fixture
|
||||||
|
KB, the real grounded path — two models: the **configured** chat
|
||||||
|
model (`turbo` per `.env` — the verdict) and `lite` (the `.env` model
|
||||||
|
temporarily switched, then restored byte-identical — the house
|
||||||
|
procedure of the 2026-09-05 turbo comparison) to check the
|
||||||
|
phase-70/72 standing block, which was recorded under `lite`. The
|
||||||
|
micro-loop (`--turns 3` — the three listing turns) ran twice on
|
||||||
|
`turbo` first: 12/12 and 13/13 contract (100 %), zero caps — so the
|
||||||
|
full battery went straight to the verdict, with no copy-lever
|
||||||
|
iteration needed (the first full run on each model was green).
|
||||||
|
|
||||||
|
```
|
||||||
|
$ uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||||||
|
restore: ok in 0.03s (8 docs, 2 sources)
|
||||||
|
turn 01 | emitted=1 executed=1 cap=no defl=no | 6.80s | List the files in this directory.
|
||||||
|
turn 02 | emitted=3 executed=3 cap=no defl=no | 9.77s | List the documents you have in the …
|
||||||
|
turn 03 | emitted=7 executed=7 cap=no defl=no | 14.83s | List every document you have indexed.
|
||||||
|
turn 04 | emitted=0 executed=0 cap=no defl=no | 8.89s | Open the document …
|
||||||
|
turn 05 | emitted=1 executed=1 cap=no defl=no | 12.48s | Read …
|
||||||
|
turn 06 | emitted=1 executed=1 cap=no defl=no | 10.03s | Open the document …
|
||||||
|
turn 07 | emitted=1 executed=1 cap=no defl=no | 6.85s | Find the exact string "rbm-8842" in …
|
||||||
|
turn 08 | emitted=0 executed=0 cap=no defl=no | 7.92s | Which document has the title "Lab …
|
||||||
|
turn 09 | emitted=0 executed=0 cap=no defl=no | 10.90s | What do you know about the qwen 3.8 …
|
||||||
|
turn 10 | emitted=5 executed=4 cap=no defl=no | 10.08s | List the files in the deployments …
|
||||||
|
gate: turbo PASS turns=10 answered=10 caps=0 tool-turns=7 calls 18/19 executed (95%) contract 19/19 (100%) 2026-09-11 (wall 98.7s)
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
(same command; `.env` chat model temporarily `lite`, restored after)
|
||||||
|
turn 01 | emitted=1 executed=1 cap=no defl=no | 3.82s | List the files in this directory.
|
||||||
|
turn 02 | emitted=8 executed=5 cap=no defl=no | 10.93s | List the documents you have in the …
|
||||||
|
turn 03 | emitted=1 executed=1 cap=no defl=no | 2.25s | List every document you have indexed.
|
||||||
|
turn 04 | emitted=1 executed=0 cap=no defl=no | 3.84s | Open the document …
|
||||||
|
turn 05 | emitted=3 executed=3 cap=no defl=no | 4.99s | Read …
|
||||||
|
turn 06 | emitted=1 executed=1 cap=no defl=no | 5.64s | Open the document …
|
||||||
|
turn 07 | emitted=1 executed=1 cap=no defl=no | 2.25s | Find the exact string "rbm-8842" in …
|
||||||
|
turn 08 | emitted=1 executed=0 cap=no defl=no | 5.09s | Which document has the title "Lab …
|
||||||
|
turn 09 | emitted=0 executed=0 cap=no defl=no | 2.93s | What do you know about the qwen 3.8 …
|
||||||
|
turn 10 | emitted=1 executed=1 cap=no defl=no | 1.75s | List the files in the deployments …
|
||||||
|
gate: lite PASS turns=10 answered=10 caps=0 tool-turns=9 calls 13/18 executed (72%) contract 18/18 (100%) 2026-09-11 (wall 43.6s)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Per-turn reading.** Both runs show the designed drill-down: the
|
||||||
|
listing turns hop `ls()` → source → folder (turbo turn 03: top level
|
||||||
|
+ both sources + five leaf folders in 7 calls; lite turn 02:
|
||||||
|
`ls('homelab')` + all three homelab folders), the read turns keep the
|
||||||
|
combined `source/path` form (turbo turn 05 drills
|
||||||
|
`ls('deployments')` → `ls('deployments/quadlet')` before the read —
|
||||||
|
the "or `ls` first, then `read`" branch), the grep turn is
|
||||||
|
pattern-only, and the seeded discipline turns (04/08/09) answer from
|
||||||
|
the `<documents>` context. Zero cap hits, zero contract violations,
|
||||||
|
10/10 answered on both models. The non-executed emitted calls are all
|
||||||
|
contract-correct (the contract is 100 %): turbo's one (turn 10) is a
|
||||||
|
second call in a single reply that the one-tool-per-reply rule
|
||||||
|
discards; lite's five are the same discard class (three on turn 02)
|
||||||
|
plus the two documented copy-invariant `ALREADY_IN_CONTEXT` re-reads
|
||||||
|
of seeded documents (turns 04/08 — §4: 15/15 across every copy
|
||||||
|
variant tried). lite turn 02 also carried one exploratory
|
||||||
|
`grep(pattern='homelab')` (pattern-only — contract-correct, executed,
|
||||||
|
then the proper folder drills in the following rounds — the same
|
||||||
|
self-correcting exploration class §5 documents pre-94).
|
||||||
|
|
||||||
|
**Baseline comparison.** Against the last recorded baseline per model
|
||||||
|
(the 2026-09-06 runs for `turbo`; the phase-70/72 standing block for
|
||||||
|
`lite`):
|
||||||
|
|
||||||
|
| model | baseline wall | phase-94 wall | Δ wall | baseline contract | phase-94 contract |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| `turbo` (configured — the verdict) | 112.8–113.7 s | 98.7 s | −12.5 % to −13.2 % | 100 % (7/7, 9/9) | 100 % (19/19) |
|
||||||
|
| `lite` (phase-70/72 standing) | 40.4–40.5 s | 43.6 s | +7.7–7.9 % | 92–100 % (last: 92 / 93) | 100 % (18/18) |
|
||||||
|
|
||||||
|
Both inside the ~20 % band — `turbo` *under* its same-model baseline
|
||||||
|
despite executing MORE calls (18 vs 7–9: the drill-down adds rounds,
|
||||||
|
but per-round latency was lower on this day than on the baseline runs
|
||||||
|
— wall is endpoint-load-sensitive, which is why a band, not a point,
|
||||||
|
is the bar), `lite` +7.7 % — the modest increase the task file
|
||||||
|
expects from a few extra `ls` hops. Accuracy: contract at or above
|
||||||
|
the same-model baseline on both models; verdict PASS on both; the
|
||||||
|
executed ratios (95 % / 72 %) sit in the documented bands (§4/§5 —
|
||||||
|
discard class + app-semantics, not tool-calling errors).
|
||||||
|
|
||||||
|
**Conclusion.** The phase-94 gate is **met**: accuracy at or above
|
||||||
|
baseline (100 % contract, PASS, zero cap hits, zero contract
|
||||||
|
violations — on both models) and wall time within the 20 % band on
|
||||||
|
both models (−12.5/−13.2 % same-model `turbo`, +7.7 % `lite`
|
||||||
|
standing). The phase-72 incident class is gone under the tree: the
|
||||||
|
flood is structurally impossible (one level per call, 50-line cap)
|
||||||
|
and the listing turns drill cleanly with the folder summaries doing
|
||||||
|
the steering. No copy-lever change was made (nothing regressed to
|
||||||
|
iterate on).
|
||||||
|
|
||||||
|
**Observation for the owner (not a gate miss).** The top level now
|
||||||
|
carries the two source summaries, and on the `lite` run the two
|
||||||
|
"list everything" turns (01, 03) answered after a single `ls()` —
|
||||||
|
from the source-level lines — without drilling to the files; `turbo`
|
||||||
|
drilled (3 / 7 calls on the same turns). The gate measures
|
||||||
|
contract/verdict, which is met either way; if the owner wants
|
||||||
|
listing answers to reach the file level by default, the lever is the
|
||||||
|
`ls` description / `TOOLS_SECTION` copy (teach that a folder's file
|
||||||
|
lines appear only after drilling into it) — flagged here rather than
|
||||||
|
acted on unilaterally.
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
"""folder_summaries: sync-time lite folder summaries (phase 94, task 01)
|
||||||
|
|
||||||
|
Revision ID: 0017
|
||||||
|
Revises: 0016
|
||||||
|
Create Date: 2026-09-10
|
||||||
|
|
||||||
|
Phase 94 (``ls`` becomes a filesystem-style drill-down tree — TODO.md
|
||||||
|
L4): the LLM lists the synced projects, then drills into folders; each
|
||||||
|
level's listing shows the folder's **subtree summary** — a 1–3 sentence
|
||||||
|
plain-text description of what the folder's documents cover, generated
|
||||||
|
at sync time by the aipi ``lite`` model (the ``KB_OVERVIEW_MODE``
|
||||||
|
contract, ``app.rag.overview``: change-gated by the caller, fail-soft,
|
||||||
|
``FOLDER_SUMMARY_MODE`` marker for the E2E mock). This migration adds
|
||||||
|
the storage — ONE additive, fully reversible table (A13), no other
|
||||||
|
schema change:
|
||||||
|
|
||||||
|
* ``folder_summaries`` — one row per folder with ≥ 2 documents
|
||||||
|
(recursive count — the same set the ``ls`` count rule counts,
|
||||||
|
``00_phase.md``; the ≥ 2 rule and the prune rule are generator
|
||||||
|
policy, ``app.rag.folder_summaries``, not schema constraints):
|
||||||
|
|
||||||
|
* PK ``(source, folder_path)`` — ``source`` is VARCHAR(120) NOT NULL
|
||||||
|
(mirrors ``documents.source``); ``folder_path`` is VARCHAR(1000)
|
||||||
|
NOT NULL (mirrors ``documents.path`` — the source-relative folder
|
||||||
|
prefix). ``folder_path = ""`` is the SOURCE ROOT: the top-level
|
||||||
|
source summary (the whole source's recursive subtree).
|
||||||
|
* ``summary`` — TEXT NOT NULL: the ``lite``-written description
|
||||||
|
(never empty — the generator validates before storing).
|
||||||
|
* ``updated_at`` — TIMESTAMPTZ NOT NULL, server default now() (house
|
||||||
|
style — the ``kb_overview.updated_at`` precedent); the generator
|
||||||
|
always stamps it explicitly on upsert.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision = "0017"
|
||||||
|
down_revision = "0016"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
op.create_table(
|
||||||
|
"folder_summaries",
|
||||||
|
sa.Column("source", sa.String(length=120), primary_key=True),
|
||||||
|
sa.Column("folder_path", sa.String(length=1000), primary_key=True),
|
||||||
|
sa.Column("summary", sa.Text(), nullable=False),
|
||||||
|
sa.Column(
|
||||||
|
"updated_at",
|
||||||
|
sa.DateTime(timezone=True),
|
||||||
|
server_default=sa.func.now(),
|
||||||
|
nullable=False,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
# The table is the only 0017 artefact — dropping it leaves the 0016
|
||||||
|
# schema byte-identical (A13, fully reversible).
|
||||||
|
op.drop_table("folder_summaries")
|
||||||
+4
-4
@@ -618,10 +618,10 @@ async def chat(
|
|||||||
)
|
)
|
||||||
return
|
return
|
||||||
except Exception: # noqa: BLE001 — a tool call hit the DB mid-stream
|
except Exception: # noqa: BLE001 — a tool call hit the DB mid-stream
|
||||||
# Phase 37: tool execution (list_catalog / find_document)
|
# Phase 37: tool execution (the drill-down ``ls`` /
|
||||||
# runs inside the stream now; a mid-turn DB failure gets
|
# ``read`` / ``grep`` lookups) runs inside the stream
|
||||||
# the same structured ``error`` event as the pre-stream
|
# now; a mid-turn DB failure gets the same structured
|
||||||
# retrieval path.
|
# ``error`` event as the pre-stream retrieval path.
|
||||||
logger.exception(
|
logger.exception(
|
||||||
"chat: tool execution failed question=%r total_ms=%d",
|
"chat: tool execution failed question=%r total_ms=%d",
|
||||||
request.message,
|
request.message,
|
||||||
|
|||||||
+31
-1
@@ -41,7 +41,16 @@ decisions):
|
|||||||
(pruning covers the union; the CLI's no-prune default is unchanged);
|
(pruning covers the union; the CLI's no-prune default is unchanged);
|
||||||
5. when the import changed the KB (added + updated > 0),
|
5. when the import changed the KB (added + updated > 0),
|
||||||
``regenerate_overview`` refreshes the single ``kb_overview`` row
|
``regenerate_overview`` refreshes the single ``kb_overview`` row
|
||||||
(phase 31 trigger, best-effort inside);
|
(phase 31 trigger, best-effort inside) — and ``generate_
|
||||||
|
folder_summaries`` (phase 94, task 02) regenerates the stored folder
|
||||||
|
summaries (the drill-down ``ls``'s per-level descriptions): its gate
|
||||||
|
is the same change trigger **plus** an empty ``folder_summaries``
|
||||||
|
table (the first sync after migration 0017 — the KB may predate the
|
||||||
|
table). It is per-folder fail-soft (a ``lite`` outage keeps the
|
||||||
|
failed folders' previous rows and never flips the run to
|
||||||
|
``failed``) and only flushes — this run's own short-lived session
|
||||||
|
commits (the phase-53 convention), so a folder failure never blocks
|
||||||
|
step 6's bump;
|
||||||
6. when the import changed the KB (added + updated + pruned > 0 — the
|
6. when the import changed the KB (added + updated + pruned > 0 — the
|
||||||
saved-chat invalidation gate, phase 53 task 02: a pruned document
|
saved-chat invalidation gate, phase 53 task 02: a pruned document
|
||||||
can invalidate a saved answer that cited it, deliberately broader
|
can invalidate a saved answer that cited it, deliberately broader
|
||||||
@@ -79,6 +88,7 @@ from app.config import get_settings
|
|||||||
from app.core.auth import require_admin
|
from app.core.auth import require_admin
|
||||||
from app.core.errors import sanitize_error as _sanitize_error
|
from app.core.errors import sanitize_error as _sanitize_error
|
||||||
from app.db import SessionLocal
|
from app.db import SessionLocal
|
||||||
|
from app.rag.folder_summaries import folder_summary_table_empty, generate_folder_summaries
|
||||||
from app.rag.git_sources import effective_sources
|
from app.rag.git_sources import effective_sources
|
||||||
from app.rag.importer import ImportSummary, import_sources
|
from app.rag.importer import ImportSummary, import_sources
|
||||||
from app.rag.llm import LLMClient, check_models
|
from app.rag.llm import LLMClient, check_models
|
||||||
@@ -251,6 +261,26 @@ async def _run_sync() -> None:
|
|||||||
overview = False
|
overview = False
|
||||||
if summary.added + summary.updated > 0:
|
if summary.added + summary.updated > 0:
|
||||||
overview = await regenerate_overview(llm)
|
overview = await regenerate_overview(llm)
|
||||||
|
# Phase 94 (task 02): the folder summaries — the drill-down
|
||||||
|
# ls's per-level descriptions. Same change gate as the overview
|
||||||
|
# (added + updated > 0), plus the table-empty first-run trigger
|
||||||
|
# (the first sync after migration 0017 — the KB may have been
|
||||||
|
# imported by the CLI before the table landed). The generator is
|
||||||
|
# per-folder fail-soft (a lite outage never flips the run to
|
||||||
|
# failed) and only flushes: this run's own short-lived session
|
||||||
|
# commits (the phase-53 convention), so the step-6 bump stays
|
||||||
|
# change-gated on the KB, not on the summaries. No status-surface
|
||||||
|
# change: the stats are log-only (the detail shape is untouched).
|
||||||
|
fs_db = SessionLocal()
|
||||||
|
try:
|
||||||
|
if summary.added + summary.updated > 0 or folder_summary_table_empty(fs_db):
|
||||||
|
folder_stats = await generate_folder_summaries(fs_db, llm)
|
||||||
|
fs_db.commit()
|
||||||
|
logger.info("sync: folder_summaries stats=%s", folder_stats)
|
||||||
|
else:
|
||||||
|
logger.info("sync: folder_summaries skipped (KB unchanged)")
|
||||||
|
finally:
|
||||||
|
fs_db.close()
|
||||||
# Phase 53 (task 02): a sync that changed the KB advances the
|
# Phase 53 (task 02): a sync that changed the KB advances the
|
||||||
# sources version exactly once — the saved-chat invalidation
|
# sources version exactly once — the saved-chat invalidation
|
||||||
# marker (task 03 stamps rows against it). The gate is
|
# marker (task 03 stamps rows against it). The gate is
|
||||||
|
|||||||
@@ -135,6 +135,14 @@ class Settings(BaseSettings):
|
|||||||
#: ``app.rag.overview``). Overflow is cut at the cap and the shared
|
#: ``app.rag.overview``). Overflow is cut at the cap and the shared
|
||||||
#: ``[…truncated…]`` marker is appended (summarizer convention).
|
#: ``[…truncated…]`` marker is appended (summarizer convention).
|
||||||
overview_input_max_chars: int = 40_000
|
overview_input_max_chars: int = 40_000
|
||||||
|
#: Cap on the folder document list (path/title/first summary line per
|
||||||
|
#: row) sent to the ``lite`` folder-summary model in ONE call
|
||||||
|
#: (phase 94, ``app.rag.folder_summaries``). Overflow is cut at the
|
||||||
|
#: cap and the shared ``[…truncated…]`` marker is appended
|
||||||
|
#: (summarizer convention). Smaller than the KB-overview cap on
|
||||||
|
#: purpose: a folder sees its own subtree only — there can be
|
||||||
|
#: hundreds of folders, each summarized separately at sync time.
|
||||||
|
folder_summary_input_max_chars: int = 8_000
|
||||||
#: Hard cap on the agent tool rounds per grounded turn (phase 45,
|
#: Hard cap on the agent tool rounds per grounded turn (phase 45,
|
||||||
#: revising phase 37's per-tool budgets — owner permission
|
#: revising phase 37's per-tool budgets — owner permission
|
||||||
#: 2026-08-27, TODO L8: "allow the LLM to make as many tool calls
|
#: 2026-08-27, TODO L8: "allow the LLM to make as many tool calls
|
||||||
|
|||||||
@@ -63,6 +63,10 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
|
|||||||
(``id = 1``); every column NULL = "use the
|
(``id = 1``); every column NULL = "use the
|
||||||
default" (env value for the strings, the built-in
|
default" (env value for the strings, the built-in
|
||||||
palette for the colors — task 01).
|
palette for the colors — task 01).
|
||||||
|
* ``folder_summaries`` — one row per folder with ≥ 2 documents:
|
||||||
|
the sync-time ``lite`` summary the drill-down
|
||||||
|
``ls`` shows next to each folder (phase 94;
|
||||||
|
``folder_path = ""`` = the source root).
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -446,3 +450,51 @@ class UiSettings(Base):
|
|||||||
accent_bg: Mapped[str | None] = mapped_column(String(7), nullable=True)
|
accent_bg: Mapped[str | None] = mapped_column(String(7), nullable=True)
|
||||||
accent_ink: Mapped[str | None] = mapped_column(String(7), nullable=True)
|
accent_ink: Mapped[str | None] = mapped_column(String(7), nullable=True)
|
||||||
accent_line: Mapped[str | None] = mapped_column(String(7), nullable=True)
|
accent_line: Mapped[str | None] = mapped_column(String(7), nullable=True)
|
||||||
|
|
||||||
|
|
||||||
|
class FolderSummary(Base):
|
||||||
|
"""One sync-time folder summary (phase 94, task 01).
|
||||||
|
|
||||||
|
``ls`` is a drill-down tree (phase 94, ``00_phase.md``): the top
|
||||||
|
level lists the synced projects, each with its stored source-root
|
||||||
|
summary; drilling into a source lists its folders, each with its
|
||||||
|
stored folder summary. This table holds those summaries:
|
||||||
|
|
||||||
|
* PK ``(source, folder_path)`` — ``source`` mirrors
|
||||||
|
``documents.source`` (String(120)); ``folder_path`` mirrors
|
||||||
|
``documents.path`` (String(1000)) and is the source-relative
|
||||||
|
folder prefix. ``folder_path = ""`` is the SOURCE ROOT — the
|
||||||
|
top-level source summary (the whole source's recursive subtree).
|
||||||
|
* Rows exist only for folders with ≥ 2 documents (recursive count
|
||||||
|
— the same set the ``ls`` count rule counts): a
|
||||||
|
single-document folder is fully described by its one file line,
|
||||||
|
so no ``lite`` burn. After a changed sync, rows whose folder
|
||||||
|
dropped below 2 documents are pruned (a pruned/renamed folder's
|
||||||
|
summary would otherwise go stale); rows for folders that still
|
||||||
|
have ≥ 2 documents persist (an unchanged folder's summary is
|
||||||
|
still true). Both rules are generator policy (``app.rag.
|
||||||
|
folder_summaries``), not schema constraints.
|
||||||
|
* ``summary`` — the 1–3 sentence plain-text description the aipi
|
||||||
|
``lite`` model wrote at sync time (``FOLDER_SUMMARY_MODE``,
|
||||||
|
``app.rag.folder_summaries`` — change-gated and fail-soft like
|
||||||
|
the KB overview: an old summary is better than none).
|
||||||
|
|
||||||
|
Chat turns only READ these rows (the agent's ``ls`` output, phase
|
||||||
|
94 task 03) — generation happens at sync time only (task 02).
|
||||||
|
"""
|
||||||
|
|
||||||
|
__tablename__ = "folder_summaries"
|
||||||
|
|
||||||
|
#: Mirrors ``documents.source`` (String(120)) — PK part 1.
|
||||||
|
source: Mapped[str] = mapped_column(String(120), primary_key=True)
|
||||||
|
#: Mirrors ``documents.path`` (String(1000)) — the source-relative
|
||||||
|
#: folder prefix; ``""`` = the source root. PK part 2.
|
||||||
|
folder_path: Mapped[str] = mapped_column(String(1000), primary_key=True)
|
||||||
|
#: The lite-written 1–3 sentence description — never empty (the
|
||||||
|
#: generator validates before storing, ``app.rag.folder_summaries``).
|
||||||
|
summary: Mapped[str] = mapped_column(Text)
|
||||||
|
#: Fresh UTC stamp on every upsert (the ``kb_overview.updated_at``
|
||||||
|
#: precedent; the generator always sets it explicitly).
|
||||||
|
updated_at: Mapped[datetime] = mapped_column(
|
||||||
|
DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
|
||||||
|
)
|
||||||
|
|||||||
+432
-84
@@ -1,5 +1,6 @@
|
|||||||
"""Agent loop: the grounded-turn document tools (phase 37, task 03; the
|
"""Agent loop: the grounded-turn document tools (phase 37, task 03; the
|
||||||
harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
|
harness-aligned ``ls``/``read``/``grep`` surface, phase 70; the
|
||||||
|
drill-down tree ``ls`` + sync-time folder summaries, phase 94).
|
||||||
|
|
||||||
Probe verdict (task 01 — ``uv run python -m scripts.llm_probe --tools``
|
Probe verdict (task 01 — ``uv run python -m scripts.llm_probe --tools``
|
||||||
run live against aipi): **``probe: turbo tool_calls=supported 2026-08-26``**
|
run live against aipi): **``probe: turbo tool_calls=supported 2026-08-26``**
|
||||||
@@ -48,14 +49,41 @@ task 04):
|
|||||||
split (with its self-correction and "teach the split" refusals) is
|
split (with its self-correction and "teach the split" refusals) is
|
||||||
gone — the model's combined form is now simply correct. The
|
gone — the model's combined form is now simply correct. The
|
||||||
phase-68 A5 match/output contract rides along under the new name.
|
phase-68 A5 match/output contract rides along under the new name.
|
||||||
|
Phase 94 revision (owner permission 2026-09-10, ``TODO.md`` L4 — the
|
||||||
|
tool-surface revision recorded in the phase 94 overview ``00_phase.md``;
|
||||||
|
PLAN.md is being redone by the owner): the ``ls`` RESULT format and
|
||||||
|
``path`` semantics changed — ``ls`` is now a filesystem-style
|
||||||
|
drill-down tree (no path: the synced sources with counts + stored
|
||||||
|
source-root summaries; a source name: its top-level folders + files;
|
||||||
|
a ``source/folder`` path: that folder's subfolders + files — the
|
||||||
|
model drills one level per call instead of flooding the whole
|
||||||
|
catalog into one result), while the tool NAME and the
|
||||||
|
``read``/``grep`` contract (combined ``source/path``) are untouched.
|
||||||
2. Each tool call the model emits is executed server-side against
|
2. Each tool call the model emits is executed server-side against
|
||||||
Postgres only (no LLM, no network): ``ls`` returns the indexed
|
Postgres only (no LLM, no network): ``ls`` lists ONE level of the
|
||||||
catalog — one ``source: X | path: Y | title: Z`` line per document
|
drill-down tree (phase 94, task 03) — no ``path``: every registered
|
||||||
(phase 63: labeled fields — unambiguous for LLM parsing),
|
source (registry order; a 0-document source still lists) as
|
||||||
``GET /api/docs`` order (uncapped in v1; the UI never shows it, only
|
``{source} — {n} documents`` plus the indented stored source-root
|
||||||
the model does) — optionally scoped to one source name (a ``path``
|
summary line when one is in ``folder_summaries``; a ``path`` that is
|
||||||
argument matching no source name is a refusal; a registered source
|
a registered source name (no ``/``): that source's root folder —
|
||||||
with no indexed documents lists as ``0 documents:`` and counts) —
|
each direct subfolder `` {sub}/ — {m} documents[: {summary}]``
|
||||||
|
(the count is the subfolder's recursive subtree — every document
|
||||||
|
whose path equals the folder or starts with ``folder + "/"``, the
|
||||||
|
same set the sync-time folder summary describes — and the file
|
||||||
|
lines ``source: X | path: Y | title: Z`` (the canonical
|
||||||
|
``read``/``grep`` identity — the phase-63 labeled format,
|
||||||
|
unchanged) in path order (``GET /api/docs`` order), capped at
|
||||||
|
:data:`LS_MAX_FILE_LINES` lines + one deterministic grep-pointer
|
||||||
|
note for the rest (a 500-file folder costs 50 lines, never 500);
|
||||||
|
a ``source/folder`` ``path``: that folder's subfolders + own file
|
||||||
|
lines (the same template, ``identity = source + "/" + folder``);
|
||||||
|
a registered source with no documents lists its header line alone
|
||||||
|
(``… — 0 documents, 0 folders:`` — the old ``0 documents:``
|
||||||
|
behavior preserved in spirit) — every successful listing (top/root/
|
||||||
|
folder) counts; a ``source/…`` argument whose first segment names no
|
||||||
|
registered source is the no-source refusal (below, the segment
|
||||||
|
echoed), and a folder segment matching no indexed prefix is the
|
||||||
|
:data:`NOT_A_FOLDER` teaching (below) —
|
||||||
``read`` takes the combined ``source/path`` string, splits it at the
|
``read`` takes the combined ``source/path`` string, splits it at the
|
||||||
FIRST ``'/'`` (source names are directory basenames — they can never
|
FIRST ``'/'`` (source names are directory basenames — they can never
|
||||||
contain ``'/'``), and returns the document's **full** content
|
contain ``'/'``), and returns the document's **full** content
|
||||||
@@ -82,15 +110,19 @@ task 04):
|
|||||||
blank or non-string) → ``"read requires a string argument
|
blank or non-string) → ``"read requires a string argument
|
||||||
'path'."``; a ``grep`` without a usable ``pattern`` (missing, blank
|
'path'."``; a ``grep`` without a usable ``pattern`` (missing, blank
|
||||||
or non-string) → ``"grep requires a string argument
|
or non-string) → ``"grep requires a string argument
|
||||||
'pattern'."``; a scoped ``ls`` whose (stripped) ``path`` contains a
|
'pattern'."``; a scoped ``ls`` whose ``path``'s FIRST segment (split
|
||||||
``/`` — a document path where a source name belongs (source names
|
at the first ``/``) names no registered source — a bare unknown
|
||||||
are directory basenames and can never contain one; the 2026-09-03
|
name (the 2026-09-03 incident's ``ls(path='.')``) or the source
|
||||||
incident's ``ls(path='app/rag/importer.py')``) →
|
segment of a ``source/…`` argument (phase 94: a ``/`` now names a
|
||||||
:data:`LS_PATH_NOT_A_SOURCE`, the document-path teaching line with
|
folder, so the phase-72 document-path teaching refusal is deleted)
|
||||||
the argument echoed; a scoped ``ls`` whose ``path`` names no
|
→ :data:`NO_SOURCE_NOT_A_DIRECTORY`, the
|
||||||
registered source (no ``/`` — the incident's ``ls(path='.')``)
|
no-source refusal with the teaching parenthetical appended, the
|
||||||
→ :data:`NO_SOURCE_NOT_A_DIRECTORY`, the no-source refusal with the
|
segment echoed; a ``source/…`` argument whose folder segment
|
||||||
teaching parenthetical appended; a document
|
matches no indexed prefix (the phase-94 existence rule — some
|
||||||
|
indexed path of the source starts with ``folder + "/"``) →
|
||||||
|
:data:`NOT_A_FOLDER`, the drill-down teaching with the argument
|
||||||
|
echoed and the deepest existing ancestor's direct subfolders
|
||||||
|
listed, so the model self-corrects in the next round; a document
|
||||||
already in context (seed or previously read) →
|
already in context (seed or previously read) →
|
||||||
:data:`ALREADY_IN_CONTEXT` (phase 72, task 05 gate iteration:
|
:data:`ALREADY_IN_CONTEXT` (phase 72, task 05 gate iteration:
|
||||||
the line names the correct action — answer from the text already
|
the line names the correct action — answer from the text already
|
||||||
@@ -164,25 +196,30 @@ needs none either (the tool ran) — the policy keys on the no-calls
|
|||||||
exit only. No model participates in detection or repair: the
|
exit only. No model participates in detection or repair: the
|
||||||
registry + the fixed retry policy are the whole guardrail.
|
registry + the fixed retry policy are the whole guardrail.
|
||||||
|
|
||||||
The DB accessors (:func:`list_catalog`, :func:`list_source_names`,
|
The DB accessors — the drill-down ``ls`` (:func:`ls_top`,
|
||||||
:func:`find_document`, :func:`all_documents`) and the
|
:func:`ls_folder`; the pure grouping :func:`group_folder_listing` and
|
||||||
:func:`grep_document` line matcher are module-level functions so unit
|
the pure renderers :func:`render_ls_top` /
|
||||||
tests can monkeypatch them without a database.
|
:func:`render_folder_listing` sit next to them),
|
||||||
|
:func:`list_source_names`, :func:`find_document`,
|
||||||
|
:func:`all_documents` — and the :func:`grep_document` line matcher are
|
||||||
|
module-level functions so unit tests can monkeypatch them without a
|
||||||
|
database.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
import re
|
import re
|
||||||
from collections.abc import AsyncIterator, Sequence
|
from collections.abc import AsyncIterator, Mapping, Sequence
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from sqlalchemy import select
|
from sqlalchemy import func, select
|
||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from app.config import Settings
|
from app.config import Settings
|
||||||
from app.models import Document
|
from app.models import Document, FolderSummary
|
||||||
|
from app.rag.folder_summaries import folder_of
|
||||||
from app.rag.git_sources import effective_sources
|
from app.rag.git_sources import effective_sources
|
||||||
from app.rag.llm import (
|
from app.rag.llm import (
|
||||||
LLMClient,
|
LLMClient,
|
||||||
@@ -212,9 +249,17 @@ AGENT_TOOLS: list[dict[str, Any]] = [
|
|||||||
"function": {
|
"function": {
|
||||||
"name": "ls",
|
"name": "ls",
|
||||||
"description": (
|
"description": (
|
||||||
"List the indexed documents as `source: X | path: Y | "
|
"List the knowledge base as a tree, one level at a "
|
||||||
"title: Z` lines. Call one tool at a time — wait for "
|
"time. With no path: the synced sources — each with "
|
||||||
"this result before your next call."
|
"its document count and a summary of its contents. "
|
||||||
|
"With a source name (no '/'): that source's top-level "
|
||||||
|
"folders and files. With a `source/folder` path: that "
|
||||||
|
"folder's subfolders and files. Folder lines carry a "
|
||||||
|
"summary of what the folder contains. File lines are "
|
||||||
|
"`source: X | path: Y | title: Z` — use the combined "
|
||||||
|
"`source/path` with `read` and `grep`. Call one tool "
|
||||||
|
"at a time — wait for this result before your next "
|
||||||
|
"call."
|
||||||
),
|
),
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"type": "object",
|
"type": "object",
|
||||||
@@ -222,13 +267,11 @@ AGENT_TOOLS: list[dict[str, Any]] = [
|
|||||||
"path": {
|
"path": {
|
||||||
"type": "string",
|
"type": "string",
|
||||||
"description": (
|
"description": (
|
||||||
"Source name to list one source's documents "
|
"Optional — a source name (e.g. "
|
||||||
"(e.g. 'homelab') — a source name, not a "
|
"'homelab') to list its top level, or a "
|
||||||
"file or directory path; omit to list "
|
"`source/folder` path to drill down "
|
||||||
"every document. This is the only tool "
|
"(e.g. 'homelab/active'). Omit it to list "
|
||||||
"whose `path` is a source name — for "
|
"every source."
|
||||||
"`read` and `grep` it must be a document's "
|
|
||||||
"combined `source/path`."
|
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -338,21 +381,6 @@ UNKNOWN_TOOL = "Unknown tool."
|
|||||||
MISSING_READ_ARGS = "read requires a string argument 'path'."
|
MISSING_READ_ARGS = "read requires a string argument 'path'."
|
||||||
MISSING_SEARCH_ARGS = "grep requires a string argument 'pattern'."
|
MISSING_SEARCH_ARGS = "grep requires a string argument 'pattern'."
|
||||||
|
|
||||||
#: Teaching refusal for a scoped ``ls`` whose stripped ``path``
|
|
||||||
#: contains a ``/`` (phase 72): a source name is a directory basename
|
|
||||||
#: and can never contain one, so the argument is a document path passed
|
|
||||||
#: where a source name belongs (the 2026-09-03 incident's
|
|
||||||
#: ``ls(path='app/rag/importer.py')``). One ``{path}`` field — the
|
|
||||||
#: argument echoed; a fixed template states the correct contract
|
|
||||||
#: instead of the terse pre-phase-72 line, so the harness-prior misuse
|
|
||||||
#: self-corrects in one round.
|
|
||||||
LS_PATH_NOT_A_SOURCE = (
|
|
||||||
"'{path}' looks like a document path, not a source name. The "
|
|
||||||
"'path' argument of ls filters by source name (e.g. 'homelab') — "
|
|
||||||
"omit it to list every document, or read a document by its "
|
|
||||||
"combined 'source/path' string."
|
|
||||||
)
|
|
||||||
|
|
||||||
#: The no-source ``ls`` refusal with the teaching parenthetical
|
#: The no-source ``ls`` refusal with the teaching parenthetical
|
||||||
#: appended (phase 72): used when a stripped scope has no ``/`` and
|
#: appended (phase 72): used when a stripped scope has no ``/`` and
|
||||||
#: matches no registered source (the incident's ``ls(path='.')``). The
|
#: matches no registered source (the incident's ``ls(path='.')``). The
|
||||||
@@ -394,6 +422,29 @@ NO_DOCUMENT_DID_YOU_MEAN_MANY = (
|
|||||||
#: dropped).
|
#: dropped).
|
||||||
SUGGESTION_LIMIT = 3
|
SUGGESTION_LIMIT = 3
|
||||||
|
|
||||||
|
#: The drill-down ``ls`` file-line cap (phase 94, task 03): a folder's
|
||||||
|
#: own files list at most this many ``source: X | path: Y | title: Z``
|
||||||
|
#: lines (path order), then one deterministic grep-pointer note — a
|
||||||
|
#: 500-file folder costs the model 50 lines + the note, never 500.
|
||||||
|
#: Pinned module constant (no env var — the phase-94 TODO asks for a
|
||||||
|
#: shape change, not a knob; the constant lives next to
|
||||||
|
#: :data:`SEARCH_MAX_MATCHES`).
|
||||||
|
LS_MAX_FILE_LINES = 50
|
||||||
|
|
||||||
|
#: The NOT-A-FOLDER ``ls`` teaching refusal (phase 94, task 03):
|
||||||
|
#: a ``source/…`` argument whose folder segment matches no indexed
|
||||||
|
#: prefix (the ``00_phase.md`` existence rule — a folder exists iff
|
||||||
|
#: some indexed path starts with ``folder + "/"``; a document's own
|
||||||
|
#: path is never a folder). Phase-72 teaching style: one line, the
|
||||||
|
#: argument echoed (``{arg}``), the DEEPEST existing ancestor's name
|
||||||
|
#: (``{parent}`` — the source for a top-level miss, ``source/folder``
|
||||||
|
#: for a nested one) and its direct subfolders (``{subfolders}`` —
|
||||||
|
#: space-joined ``name/`` entries in path order, so the model
|
||||||
|
#: self-corrects in the next round; ``none`` when the ancestor has no
|
||||||
|
#: subfolders). Still a refusal: it counts in nothing and consumes a
|
||||||
|
#: round (no silent argument normalization).
|
||||||
|
NOT_A_FOLDER = "'{arg}' is not a folder — {parent} has: {subfolders}"
|
||||||
|
|
||||||
#: The harness-owned recovery line (phase 71, task 03) — folded into the
|
#: The harness-owned recovery line (phase 71, task 03) — folded into the
|
||||||
#: ORIGINAL single system message of the one bounded recovery request
|
#: ORIGINAL single system message of the one bounded recovery request
|
||||||
#: (``system_prompt + "\n" + CORRECTION_INSTRUCTION``; provider-safe,
|
#: (``system_prompt + "\n" + CORRECTION_INSTRUCTION``; provider-safe,
|
||||||
@@ -507,20 +558,6 @@ def plain_form(pattern: str) -> str:
|
|||||||
return p.strip()
|
return p.strip()
|
||||||
|
|
||||||
|
|
||||||
def list_catalog(db: Session) -> list[tuple[str, str, str]]:
|
|
||||||
"""Every indexed document as ``(source, path, title)``.
|
|
||||||
|
|
||||||
Ordered by ``(source, path)`` — the same order as ``GET /api/docs``.
|
|
||||||
Module-level (not a method) so unit tests can monkeypatch it.
|
|
||||||
"""
|
|
||||||
rows = db.execute(
|
|
||||||
select(Document.source, Document.path, Document.title).order_by(
|
|
||||||
Document.source, Document.path
|
|
||||||
)
|
|
||||||
).all()
|
|
||||||
return [(source, path, title) for source, path, title in rows]
|
|
||||||
|
|
||||||
|
|
||||||
def list_source_names(db: Session) -> list[str]:
|
def list_source_names(db: Session) -> list[str]:
|
||||||
"""Every registered source name, deduped, in registry order.
|
"""Every registered source name, deduped, in registry order.
|
||||||
|
|
||||||
@@ -546,6 +583,286 @@ def list_source_names(db: Session) -> list[str]:
|
|||||||
return names
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
#: The drill-down ``ls`` fetchers (phase 94, task 03) — module-level so
|
||||||
|
#: unit tests can monkeypatch them without a database (the house style:
|
||||||
|
#: :func:`ls_top` / :func:`ls_folder` compose them; the pure grouping
|
||||||
|
#: :func:`group_folder_listing` and renderers
|
||||||
|
#: :func:`render_ls_top` / :func:`render_folder_listing` sit below).
|
||||||
|
|
||||||
|
|
||||||
|
def _source_document_counts(db: Session) -> list[tuple[str, int]]:
|
||||||
|
"""``(source, document count)`` per indexed source (one grouped
|
||||||
|
query — the top-level ``ls`` counts, phase 94 task 03)."""
|
||||||
|
return [
|
||||||
|
(source, count)
|
||||||
|
for source, count in db.execute(
|
||||||
|
select(Document.source, func.count()).group_by(Document.source)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _source_root_summaries(db: Session) -> list[tuple[str, str]]:
|
||||||
|
"""The stored source-root summaries ``(source, summary)``
|
||||||
|
(``folder_path = ""`` — the top-level ``ls`` indented lines, phase
|
||||||
|
94 task 03; absent when never generated)."""
|
||||||
|
return [
|
||||||
|
(source, summary)
|
||||||
|
for source, summary in db.execute(
|
||||||
|
select(FolderSummary.source, FolderSummary.summary).where(
|
||||||
|
FolderSummary.folder_path == ""
|
||||||
|
)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str]]:
|
||||||
|
"""``(path, title)`` of every document under *source*, ordered by
|
||||||
|
``path`` — the one bounded fetch a folder drill level lists (phase
|
||||||
|
94 task 03; one source's paths, not the whole KB)."""
|
||||||
|
return [
|
||||||
|
(path, title)
|
||||||
|
for path, title in db.execute(
|
||||||
|
select(Document.path, Document.title)
|
||||||
|
.where(Document.source == source)
|
||||||
|
.order_by(Document.path)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _source_folder_summaries(db: Session, source: str) -> dict[str, str]:
|
||||||
|
"""The stored folder summaries ``{folder_path: summary}`` of one
|
||||||
|
source (phase 94 task 03; includes the ``""`` source-root row when
|
||||||
|
stored)."""
|
||||||
|
return {
|
||||||
|
folder_path: summary
|
||||||
|
for folder_path, summary in db.execute(
|
||||||
|
select(FolderSummary.folder_path, FolderSummary.summary).where(
|
||||||
|
FolderSummary.source == source
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def ls_top(db: Session) -> list[tuple[str, int, str | None]]:
|
||||||
|
"""The top level of the drill-down ``ls`` (phase 94, task 03).
|
||||||
|
|
||||||
|
Every registered source in :func:`list_source_names` order (the
|
||||||
|
registry is the source of truth — a registered source with 0
|
||||||
|
indexed documents still lists, the phase-70/72 invariant) as
|
||||||
|
``(name, recursive_document_count, source_root_summary)``: the
|
||||||
|
count is the source's whole subtree (all of its documents — the
|
||||||
|
same set its stored summary describes) and the summary is the
|
||||||
|
stored ``folder_summaries`` row for ``(source, "")`` (the source
|
||||||
|
root, phase 94 task 01) or ``None`` when absent. Module-level so
|
||||||
|
unit tests can monkeypatch the fetchers.
|
||||||
|
"""
|
||||||
|
names = list_source_names(db)
|
||||||
|
if not names:
|
||||||
|
return []
|
||||||
|
counts = dict(_source_document_counts(db))
|
||||||
|
summaries = dict(_source_root_summaries(db))
|
||||||
|
return [(name, counts.get(name, 0), summaries.get(name)) for name in names]
|
||||||
|
|
||||||
|
|
||||||
|
def group_folder_listing(
|
||||||
|
source: str,
|
||||||
|
folder: str,
|
||||||
|
rows: Sequence[tuple[str, str]],
|
||||||
|
summaries: Mapping[str, str],
|
||||||
|
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
|
||||||
|
"""One level of the drill-down tree (phase 94, task 03) — pure.
|
||||||
|
|
||||||
|
Given *rows* — the source's ``(path, title)`` pairs in catalog
|
||||||
|
(path) order — and *summaries* (the source's stored
|
||||||
|
``folder_summaries`` rows: ``folder_path → summary``), the folder
|
||||||
|
level *folder* (source-relative; ``""`` = the source root):
|
||||||
|
|
||||||
|
* **(a) direct subfolders** — the folders whose parent is exactly
|
||||||
|
*folder*, in path order, each
|
||||||
|
``(sub_path_relative_to_source, recursive_count,
|
||||||
|
summary_or_None)``. A folder is a slash-boundary prefix of at
|
||||||
|
least one indexed path (the ``00_phase.md`` existence rule: a
|
||||||
|
folder ``F`` exists ⟺ some path starts with ``F + "/"`` — a
|
||||||
|
document's OWN path is never a folder); the count is the
|
||||||
|
folder's recursive subtree — every path equal to the folder or
|
||||||
|
starting with ``folder + "/"`` (the same set the sync-time
|
||||||
|
folder summary describes, phase 94 task 01 — one concept end to
|
||||||
|
end).
|
||||||
|
* **(b) direct file lines** — the documents whose folder (the
|
||||||
|
prefix before the last ``/`` —
|
||||||
|
:func:`app.rag.folder_summaries.folder_of`, the shared notion) IS
|
||||||
|
*folder*, in path order (catalog order — the same order
|
||||||
|
``GET /api/docs`` serves), as ``(source, path, title)`` triples
|
||||||
|
— the canonical ``read``/``grep`` identity, capped at
|
||||||
|
:data:`LS_MAX_FILE_LINES` (the rest fold into the renderer's
|
||||||
|
note; a 500-file folder never costs 500 lines).
|
||||||
|
* **(c) the TOTAL direct-file count** — pre-cap, for the note.
|
||||||
|
|
||||||
|
Pure (no I/O) — unit tests drive the grouping without a database;
|
||||||
|
:func:`ls_folder` is the DB-composing wrapper.
|
||||||
|
"""
|
||||||
|
# The source's existing folders: every slash-boundary prefix of an
|
||||||
|
# indexed path (the existence rule's candidate set — a folder is
|
||||||
|
# present iff at least one path starts with ``folder + "/"``).
|
||||||
|
folders: set[str] = set()
|
||||||
|
for path, _title in rows:
|
||||||
|
f = folder_of(path)
|
||||||
|
while f:
|
||||||
|
folders.add(f)
|
||||||
|
f = folder_of(f)
|
||||||
|
# The recursive count per folder — the ``path == folder`` arm (a
|
||||||
|
# document sharing a folder's name) plus the ``startswith
|
||||||
|
# folder + "/"`` arm (the folder's true descendants), one pass per
|
||||||
|
# document.
|
||||||
|
counts: dict[str, int] = {f: 0 for f in folders}
|
||||||
|
for path, _title in rows:
|
||||||
|
if path in folders:
|
||||||
|
counts[path] += 1
|
||||||
|
f = folder_of(path)
|
||||||
|
while f:
|
||||||
|
counts[f] += 1
|
||||||
|
f = folder_of(f)
|
||||||
|
subfolders = [
|
||||||
|
(g, counts[g], summaries.get(g))
|
||||||
|
for g in sorted(g for g in folders if folder_of(g) == folder)
|
||||||
|
]
|
||||||
|
files = [
|
||||||
|
(source, path, title)
|
||||||
|
for path, title in rows
|
||||||
|
if folder_of(path) == folder
|
||||||
|
]
|
||||||
|
return subfolders, files[:LS_MAX_FILE_LINES], len(files)
|
||||||
|
|
||||||
|
|
||||||
|
def ls_folder(
|
||||||
|
db: Session, source: str, folder: str
|
||||||
|
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
|
||||||
|
"""One folder level of the drill-down ``ls`` (phase 94, task 03).
|
||||||
|
|
||||||
|
The source's document rows (:func:`_source_document_rows`) and
|
||||||
|
stored folder summaries (:func:`_source_folder_summaries`) through
|
||||||
|
:func:`group_folder_listing` — the pure grouping the unit tests
|
||||||
|
drive directly. ``folder = ""`` is the source root. Module-level
|
||||||
|
so unit tests can monkeypatch the fetchers.
|
||||||
|
"""
|
||||||
|
return group_folder_listing(
|
||||||
|
source,
|
||||||
|
folder,
|
||||||
|
_source_document_rows(db, source),
|
||||||
|
_source_folder_summaries(db, source),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
|
||||||
|
"""The phase-94 folder-existence rule (``00_phase.md``), pure.
|
||||||
|
|
||||||
|
Folder *folder* (source-relative) under a registered source
|
||||||
|
exists ⟺ ``folder == ""`` OR some indexed path starts with
|
||||||
|
``folder + "/"`` — a document's OWN path is never a folder
|
||||||
|
(nothing starts with ``path + "/"``), so ``ls`` of a file path
|
||||||
|
refuses with :data:`NOT_A_FOLDER` rather than listing.
|
||||||
|
"""
|
||||||
|
if not folder:
|
||||||
|
return True
|
||||||
|
prefix = folder + "/"
|
||||||
|
return any(path.startswith(prefix) for path, _title in rows)
|
||||||
|
|
||||||
|
|
||||||
|
def _deepest_existing_ancestor(
|
||||||
|
rows: Sequence[tuple[str, str]], folder: str
|
||||||
|
) -> str:
|
||||||
|
"""The deepest EXISTING folder prefix of a missing *folder* (pure).
|
||||||
|
|
||||||
|
The :data:`NOT_A_FOLDER` refusal's teaching context: the argument's
|
||||||
|
segments are walked from the top; the walk stops at the first
|
||||||
|
segment that is no folder, so the returned prefix is the deepest
|
||||||
|
existing ancestor (``""`` = the source root when the first segment
|
||||||
|
is already the miss) and its direct subfolders are the bounded
|
||||||
|
self-correction list the refusal prints.
|
||||||
|
"""
|
||||||
|
parent = ""
|
||||||
|
for part in folder.split("/"):
|
||||||
|
candidate = f"{parent}/{part}" if parent else part
|
||||||
|
if not _folder_exists_in(rows, candidate):
|
||||||
|
break
|
||||||
|
parent = candidate
|
||||||
|
return parent
|
||||||
|
|
||||||
|
|
||||||
|
def render_ls_top(entries: Sequence[tuple[str, int, str | None]]) -> str:
|
||||||
|
"""The top-level ``ls`` result (phase 94, task 03) — the pinned
|
||||||
|
template.
|
||||||
|
|
||||||
|
``{N} sources:`` — the line alone when the registry is empty (the
|
||||||
|
old ``0 documents:`` behavior preserved in spirit) — then, when at
|
||||||
|
least one source is registered, a blank line and one block per
|
||||||
|
source in registry order: ``{source} — {n} documents`` plus the
|
||||||
|
indented `` {summary}`` line ONLY when the source-root summary is
|
||||||
|
stored (absent → the count line alone, no placeholder) — with NO
|
||||||
|
blank line between blocks.
|
||||||
|
"""
|
||||||
|
lines = [f"{len(entries)} sources:"]
|
||||||
|
if entries:
|
||||||
|
lines.append("")
|
||||||
|
for source, count, summary in entries:
|
||||||
|
lines.append(f"{source} — {count} documents")
|
||||||
|
if summary:
|
||||||
|
lines.append(f" {summary}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def render_folder_listing(
|
||||||
|
identity: str,
|
||||||
|
subfolders: Sequence[tuple[str, int, str | None]],
|
||||||
|
files: Sequence[tuple[str, str, str]],
|
||||||
|
total_files: int,
|
||||||
|
) -> str:
|
||||||
|
"""One folder level of the drill-down ``ls`` (phase 94, task 03) —
|
||||||
|
the pinned template.
|
||||||
|
|
||||||
|
The header ``{identity} — {n_files} documents, {n_folders}
|
||||||
|
folders:`` (``n_files`` = *total_files*, the PRE-cap direct-file
|
||||||
|
count; ``identity`` is the source name at the root level and
|
||||||
|
``source/folder`` below it), then — when the level carries anything
|
||||||
|
below the header — a blank line, the 2-space-indented subfolder
|
||||||
|
lines `` {sub}/ — {m} documents`` in path order (``: {summary}``
|
||||||
|
appended ONLY when the subfolder's summary is stored), a blank
|
||||||
|
line, the file lines in EXACTLY the existing
|
||||||
|
``source: X | path: Y | title: Z`` format (the canonical
|
||||||
|
``read``/``grep`` identity — unchanged), and the cap note
|
||||||
|
``…and {hidden} more documents in this folder — use grep
|
||||||
|
(pattern) to find a specific one.`` ONLY when the folder's own
|
||||||
|
files outnumber :data:`LS_MAX_FILE_LINES` (*files* arrives capped;
|
||||||
|
*total_files* carries the pre-cap count). A header with no
|
||||||
|
subfolders and no files — a registered source with no documents —
|
||||||
|
is the header line alone (``… — 0 documents, 0 folders:``).
|
||||||
|
"""
|
||||||
|
header = f"{identity} — {total_files} documents, {len(subfolders)} folders:"
|
||||||
|
if not subfolders and not files:
|
||||||
|
return header
|
||||||
|
body: list[str] = []
|
||||||
|
if subfolders:
|
||||||
|
body.append("")
|
||||||
|
for sub, count, summary in subfolders:
|
||||||
|
line = f" {sub}/ — {count} documents"
|
||||||
|
if summary:
|
||||||
|
line += f": {summary}"
|
||||||
|
body.append(line)
|
||||||
|
if files or total_files > len(files):
|
||||||
|
body.append("")
|
||||||
|
body.extend(
|
||||||
|
f"source: {source} | path: {path} | title: {title}"
|
||||||
|
for source, path, title in files
|
||||||
|
)
|
||||||
|
hidden = total_files - len(files)
|
||||||
|
if hidden > 0:
|
||||||
|
body.append(
|
||||||
|
f"…and {hidden} more documents in this folder — use grep "
|
||||||
|
"(pattern) to find a specific one."
|
||||||
|
)
|
||||||
|
return "\n".join([header, *body])
|
||||||
|
|
||||||
|
|
||||||
def find_document(db: Session, source: str, path: str) -> Document | None:
|
def find_document(db: Session, source: str, path: str) -> Document | None:
|
||||||
"""The indexed document at ``(source, path)``, or ``None``.
|
"""The indexed document at ``(source, path)``, or ``None``.
|
||||||
|
|
||||||
@@ -704,30 +1021,61 @@ def _execute_tool(
|
|||||||
phase 70).
|
phase 70).
|
||||||
"""
|
"""
|
||||||
if call.name == "ls":
|
if call.name == "ls":
|
||||||
|
# Phase 94 (task 03): the drill-down tree — one level per call
|
||||||
|
# (the owner-permitted tool-surface revision; the old
|
||||||
|
# whole-catalog listing is gone). A successful listing at ANY
|
||||||
|
# level (top/root/folder) counts; a refusal counts in nothing
|
||||||
|
# and consumes a round like every refusal.
|
||||||
raw_path = call.arguments.get("path")
|
raw_path = call.arguments.get("path")
|
||||||
scope = raw_path.strip() if isinstance(raw_path, str) else ""
|
scope = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||||
rows = list_catalog(db)
|
if not scope:
|
||||||
if scope:
|
# The top level: the synced sources, registry order, each
|
||||||
if "/" in scope:
|
# with its recursive document count and its stored
|
||||||
# A source name (a directory basename) can never
|
# source-root summary (``None`` → no indented line).
|
||||||
# contain '/' — this is a document path where a source
|
holder.tool_calls += 1
|
||||||
# name belongs (phase 72): teach the contract; no
|
return render_ls_top(ls_top(db))
|
||||||
# registry lookup needed, counts in nothing, consumes
|
source, _, rest = scope.partition("/")
|
||||||
# a round like every refusal.
|
if source not in list_source_names(db):
|
||||||
return LS_PATH_NOT_A_SOURCE.format(path=scope)
|
# The no-source refusal with the teaching parenthetical
|
||||||
if scope not in list_source_names(db):
|
# (phase 72, the prefix byte-identical to the pre-phase-72
|
||||||
# The no-source refusal with the teaching parenthetical
|
# line): the FIRST segment is the source candidate — a bare
|
||||||
# (phase 72) — the prefix byte-identical to the
|
# unknown name (the incident's ``ls(path='.')``) or the
|
||||||
# pre-phase-72 line; counts in nothing, consumes a
|
# source segment of a ``source/…`` argument (phase 94: a
|
||||||
# round like every refusal.
|
# ``/`` now names a folder, so the phase-72 document-path
|
||||||
return NO_SOURCE_NOT_A_DIRECTORY.format(scope=scope)
|
# teaching is deleted) — the segment echoed; counts in
|
||||||
rows = [row for row in rows if row[0] == scope]
|
# nothing, consumes a round like every refusal.
|
||||||
listing = f"{len(rows)} documents:\n" + "\n".join(
|
return NO_SOURCE_NOT_A_DIRECTORY.format(scope=source)
|
||||||
f"source: {source} | path: {path} | title: {title}"
|
if not rest:
|
||||||
for source, path, title in rows
|
# The source's ROOT folder: subfolders + own file lines
|
||||||
)
|
# (capped + note) — the pinned template; a registered
|
||||||
|
# source with no documents lists its header line alone
|
||||||
|
# (``… — 0 documents, 0 folders:`` — the old
|
||||||
|
# ``0 documents:`` behavior preserved in spirit).
|
||||||
|
subfolders, files, total = ls_folder(db, source, "")
|
||||||
|
holder.tool_calls += 1
|
||||||
|
return render_folder_listing(source, subfolders, files, total)
|
||||||
|
rows = _source_document_rows(db, source)
|
||||||
|
summaries = _source_folder_summaries(db, source)
|
||||||
|
if not _folder_exists_in(rows, rest):
|
||||||
|
# The NOT-A-FOLDER teaching (phase 94, task 03): the
|
||||||
|
# argument echoed, the DEEPEST existing ancestor's name and
|
||||||
|
# its direct subfolders listed (bounded — the parent's own
|
||||||
|
# listing, so no new flood path), so the model
|
||||||
|
# self-corrects in the next round; counts in nothing,
|
||||||
|
# consumes a round like every refusal.
|
||||||
|
parent = _deepest_existing_ancestor(rows, rest)
|
||||||
|
parent_subs = group_folder_listing(source, parent, rows, summaries)[0]
|
||||||
|
return NOT_A_FOLDER.format(
|
||||||
|
arg=scope,
|
||||||
|
parent=source if not parent else f"{source}/{parent}",
|
||||||
|
subfolders=" ".join(f"{sub}/" for sub, _c, _s in parent_subs)
|
||||||
|
or "none",
|
||||||
|
)
|
||||||
|
# The folder level: the same template as the root, identity =
|
||||||
|
# source + "/" + folder.
|
||||||
|
subfolders, files, total = group_folder_listing(source, rest, rows, summaries)
|
||||||
holder.tool_calls += 1
|
holder.tool_calls += 1
|
||||||
return listing
|
return render_folder_listing(f"{source}/{rest}", subfolders, files, total)
|
||||||
if call.name == "read":
|
if call.name == "read":
|
||||||
raw_path = call.arguments.get("path")
|
raw_path = call.arguments.get("path")
|
||||||
arg = raw_path.strip() if isinstance(raw_path, str) else ""
|
arg = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||||
|
|||||||
@@ -0,0 +1,391 @@
|
|||||||
|
"""Folder summary generator (phase 94, task 01).
|
||||||
|
|
||||||
|
``ls`` becomes a drill-down tree (phase 94, ``00_phase.md``): the LLM
|
||||||
|
lists the synced projects, then drills into folders. Each level's
|
||||||
|
listing shows the folder's **subtree summary** — a 1–3 sentence
|
||||||
|
plain-text description of what the folder's documents cover, generated
|
||||||
|
at SYNC time by the aipi ``lite`` model (the ``KB_OVERVIEW_MODE``
|
||||||
|
contract of ``app.rag.overview``: change-gated by the caller,
|
||||||
|
fail-soft — an old summary is better than none — and E2E-mockable via
|
||||||
|
the ``FOLDER_SUMMARY_MODE`` system-prompt marker).
|
||||||
|
|
||||||
|
ONE concept end to end (:func:`group_by_folder`): a folder row's
|
||||||
|
documents are its **recursive subtree** — every document whose path
|
||||||
|
equals the folder or starts with ``folder + "/"`` — exactly the set the
|
||||||
|
``ls`` count rule (``00_phase.md``) counts. A document under ``a/b/``
|
||||||
|
therefore contributes to the ``a``, ``a/b``, and ``""`` (source root)
|
||||||
|
groups alike: each level's listing shows its own accurate subtree
|
||||||
|
summary, and the number next to a folder is the number of documents its
|
||||||
|
summary describes.
|
||||||
|
|
||||||
|
Storage: ``folder_summaries`` (migration 0017) — PK
|
||||||
|
``(source, folder_path)``; ``folder_path = ""`` is the SOURCE ROOT
|
||||||
|
(the top-level source summary). Rows exist only for folders with
|
||||||
|
≥ 2 documents (the :data:`MIN_DOCS_PER_FOLDER` rule — a
|
||||||
|
single-document folder is fully described by its one file line, so no
|
||||||
|
``lite`` burn); after a changed sync, rows whose folder dropped below
|
||||||
|
2 documents are pruned (a pruned/renamed folder's summary would
|
||||||
|
otherwise go stale), while rows for folders that still have
|
||||||
|
≥ 2 documents persist (an unchanged folder's summary is still true).
|
||||||
|
|
||||||
|
Chat turns never generate folder summaries — the agent's ``ls`` output
|
||||||
|
(phase 94, task 03) only reads the stored rows. Generation is the
|
||||||
|
caller's job at sync time (phase 94, task 02), and :func:`generate_
|
||||||
|
folder_summaries` only flushes — the sync path owns the transaction
|
||||||
|
(the phase-53 ``bump_sources_version`` convention).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.config import Settings, get_settings
|
||||||
|
from app.models import Document, FolderSummary
|
||||||
|
from app.rag.llm import LLMError
|
||||||
|
from app.rag.retriever import TRUNCATION_MARKER
|
||||||
|
|
||||||
|
logger = logging.getLogger("app.rag.folder_summaries")
|
||||||
|
|
||||||
|
#: System-prompt marker for folder-summary generation — the E2E mock
|
||||||
|
#: LLM keys on it (same convention as ``SUMMARY_MODE`` /
|
||||||
|
#: ``KB_OVERVIEW_MODE``, PLAN §6).
|
||||||
|
FOLDER_SUMMARY_MODE = "FOLDER_SUMMARY_MODE"
|
||||||
|
|
||||||
|
#: Locked instruction for the ``lite`` model (phase 94): the folder
|
||||||
|
#: summary is the drill-down ``ls``'s per-level picture, so it must be
|
||||||
|
#: a short plain-text description of what the folder's documents
|
||||||
|
#: cover — 1–3 sentences (a folder is a skim, not a read: the document
|
||||||
|
#: summary's 3–6 sentences would blow up a 20-folder listing),
|
||||||
|
#: strictly grounded in the listed titles/paths/summary lines.
|
||||||
|
FOLDER_SUMMARY_INSTRUCTION = (
|
||||||
|
"From the document list below, write a 1-3 sentence plain-text "
|
||||||
|
"summary of what this folder's documents cover, in natural "
|
||||||
|
"language. Do not use markdown. Do not invent anything that is not "
|
||||||
|
"in the list."
|
||||||
|
)
|
||||||
|
|
||||||
|
#: Full system prompt: marker first (the mock's key), then the
|
||||||
|
#: instruction (the ``app.rag.overview.SYSTEM_PROMPT`` shape).
|
||||||
|
SYSTEM_PROMPT = f"{FOLDER_SUMMARY_MODE}: {FOLDER_SUMMARY_INSTRUCTION}"
|
||||||
|
|
||||||
|
#: First line of the user message — the folder the summary describes:
|
||||||
|
#: ``<source>`` for the source root, ``<source>/<folder_path>`` for a
|
||||||
|
#: folder. The deterministic E2E mock keys on it to name the folder in
|
||||||
|
#: its canned reply (``tests/e2e/mock_llm.py``), so the stored row is
|
||||||
|
#: a pure function of the request.
|
||||||
|
FOLDER_HEADER_PREFIX = "Folder: "
|
||||||
|
|
||||||
|
#: A folder is summarized only while its recursive subtree holds at
|
||||||
|
#: least this many documents — a single-document folder is fully
|
||||||
|
#: described by its one file line, so no ``lite`` burn (phase 94
|
||||||
|
#: ``00_phase.md`` scope rule; the prune rule applies the same count).
|
||||||
|
MIN_DOCS_PER_FOLDER = 2
|
||||||
|
|
||||||
|
#: One document row for the grouping/prompting:
|
||||||
|
#: ``(source, path, title, summary)`` — the ``app.rag.overview``
|
||||||
|
#: ``build_overview_prompt`` row shape (``summary`` is the stored
|
||||||
|
#: lite-written text or ``None`` for markdown docs / the fail-soft
|
||||||
|
#: path).
|
||||||
|
DocRow = tuple[str, str, str, str | None]
|
||||||
|
|
||||||
|
|
||||||
|
class FolderSummaryLLM(Protocol):
|
||||||
|
"""The one-shot chat surface the folder-summary generator needs.
|
||||||
|
|
||||||
|
:class:`app.rag.llm.LLMClient` satisfies it; unit tests pass a
|
||||||
|
duck-typed fake (``chat`` + ``settings``) instead — same pattern as
|
||||||
|
the overview's ``OverviewLLM`` protocol.
|
||||||
|
"""
|
||||||
|
|
||||||
|
settings: Settings
|
||||||
|
|
||||||
|
async def chat(
|
||||||
|
self, messages: list[dict[str, str]], model: str | None = None
|
||||||
|
) -> str: ...
|
||||||
|
|
||||||
|
|
||||||
|
def folder_of(path: str) -> str:
|
||||||
|
"""The directory prefix before the last ``/`` (``""`` for root-level files).
|
||||||
|
|
||||||
|
The single notion of "which folder owns this document" used by the
|
||||||
|
whole module: ``folder_of("a.md") == ""``,
|
||||||
|
``folder_of("a/b.md") == "a"``, ``folder_of("a/b/c.md") == "a/b"``.
|
||||||
|
Iterating :func:`folder_of` over its own result walks a path's
|
||||||
|
folder prefixes from nearest to farthest, ending at ``""``.
|
||||||
|
"""
|
||||||
|
idx = path.rfind("/")
|
||||||
|
return path[:idx] if idx >= 0 else ""
|
||||||
|
|
||||||
|
|
||||||
|
def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow]]:
|
||||||
|
"""Group document rows by the folders their recursive subtree fills.
|
||||||
|
|
||||||
|
The ONE concept of this module, documented in the module docstring:
|
||||||
|
a folder row's documents are its **recursive subtree** — every
|
||||||
|
document whose path equals the folder or starts with
|
||||||
|
``folder + "/"`` — exactly the set the ``ls`` count rule counts.
|
||||||
|
Each row therefore lands in the ``""`` (source root) group AND in
|
||||||
|
the group of every folder prefix of its path: a doc under
|
||||||
|
``a/b/`` contributes to the ``a``, ``a/b``, and ``""`` groups.
|
||||||
|
Per source the candidate rows are thus ``""`` (all of the source's
|
||||||
|
documents — the top-level source summary) plus every distinct
|
||||||
|
folder prefix of an indexed path.
|
||||||
|
|
||||||
|
*rows* are ``(source, path, title, summary)`` tuples (the
|
||||||
|
:data:`DocRow` shape, e.g. straight from the ``documents``
|
||||||
|
catalogue query). Returns ``{(source, folder_path): [rows]}`` —
|
||||||
|
group lists keep the input (catalogue) order, so downstream prompt
|
||||||
|
building is deterministic. Groups of ANY size (≥ 1) are returned;
|
||||||
|
the ≥ 2 :data:`MIN_DOCS_PER_FOLDER` rule is applied by
|
||||||
|
:func:`generate_folder_summaries`, not here.
|
||||||
|
|
||||||
|
"""
|
||||||
|
# Pass 1: the distinct TRUE folder prefixes of the catalogue — a
|
||||||
|
# folder is a slash-boundary prefix of at least one indexed path
|
||||||
|
# (the ``00_phase.md`` candidate definition: ``""`` + every distinct
|
||||||
|
# folder prefix, per source).
|
||||||
|
folders: set[tuple[str, str]] = set()
|
||||||
|
for row in rows:
|
||||||
|
source, path = row[0], row[1]
|
||||||
|
folder = folder_of(path)
|
||||||
|
while folder:
|
||||||
|
folders.add((source, folder))
|
||||||
|
folder = folder_of(folder)
|
||||||
|
|
||||||
|
# Pass 2: every row lands in the source-root group, in the group of
|
||||||
|
# every ancestor folder (the ``startswith folder + "/"`` arm of the
|
||||||
|
# count rule), and — when its path IS one of the source's folder
|
||||||
|
# prefixes (a file sharing its name with a directory) — in that
|
||||||
|
# folder's group too (the ``path == folder`` arm). A row therefore
|
||||||
|
# belongs to its folder's group iff its path equals the folder or
|
||||||
|
# starts with ``folder + "/"`` — exactly the set the ``ls`` count
|
||||||
|
# rule counts, for EVERY group.
|
||||||
|
groups: dict[tuple[str, str], list[DocRow]] = {}
|
||||||
|
for row in rows:
|
||||||
|
source, path = row[0], row[1]
|
||||||
|
groups.setdefault((source, ""), []).append(row)
|
||||||
|
if (source, path) in folders:
|
||||||
|
groups.setdefault((source, path), []).append(row)
|
||||||
|
folder = folder_of(path)
|
||||||
|
while folder:
|
||||||
|
groups.setdefault((source, folder), []).append(row)
|
||||||
|
folder = folder_of(folder)
|
||||||
|
return groups
|
||||||
|
|
||||||
|
|
||||||
|
def _first_summary_line(summary: str | None) -> str:
|
||||||
|
"""First line of a stored summary, stripped; ``''`` when absent.
|
||||||
|
|
||||||
|
The stored summary (phase 30) ends in the code-appended
|
||||||
|
``Source: …`` pointer line, so its first line is the model-written
|
||||||
|
lead sentence — the best one-line picture of the document for the
|
||||||
|
folder summary. Blank/whitespace-only summaries yield ``''`` as
|
||||||
|
well (the ``app.rag.overview`` helper, mirrored locally so each
|
||||||
|
lite-mode module stays self-contained).
|
||||||
|
"""
|
||||||
|
if not summary:
|
||||||
|
return ""
|
||||||
|
for line in summary.splitlines():
|
||||||
|
stripped = line.strip()
|
||||||
|
if stripped:
|
||||||
|
return stripped
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _folder_label(source: str, folder_path: str) -> str:
|
||||||
|
"""The user-message folder identifier (the ``FOLDER_HEADER_PREFIX`` tail).
|
||||||
|
|
||||||
|
``<source>`` for the source root (``folder_path = ""``),
|
||||||
|
``<source>/<folder_path>`` for a folder — the canonical folder
|
||||||
|
identity the E2E mock echoes into its canned reply.
|
||||||
|
"""
|
||||||
|
return source if not folder_path else f"{source}/{folder_path}"
|
||||||
|
|
||||||
|
|
||||||
|
def build_folder_summary_prompt(
|
||||||
|
source: str,
|
||||||
|
folder_path: str,
|
||||||
|
docs: Sequence[DocRow],
|
||||||
|
max_chars: int | None = None,
|
||||||
|
) -> tuple[str, str]:
|
||||||
|
"""The ``(system, user)`` message pair for one folder-summary call.
|
||||||
|
|
||||||
|
* ``system`` — :data:`SYSTEM_PROMPT`: the ``FOLDER_SUMMARY_MODE``
|
||||||
|
marker + the locked instruction.
|
||||||
|
* ``user`` — the folder header line
|
||||||
|
(``FOLDER_HEADER_PREFIX + <source>[/<folder_path>]`` — the line
|
||||||
|
the E2E mock parses to name the folder), then one line per
|
||||||
|
document, ``path — title — {first line of summary}``, joined
|
||||||
|
with newlines, in the given (catalogue) order. The summary field
|
||||||
|
is omitted when the document has none (markdown docs and the
|
||||||
|
fail-soft path — no dangling dash, the overview convention). The
|
||||||
|
whole message is capped at *max_chars* (default
|
||||||
|
``BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS``): overflow is cut exactly
|
||||||
|
at the cap and the shared ``[…truncated…]`` marker is appended on
|
||||||
|
its own line, so the model never sees more than the cap and the
|
||||||
|
cut is visible (summarizer convention, phase 30).
|
||||||
|
"""
|
||||||
|
lines = [FOLDER_HEADER_PREFIX + _folder_label(source, folder_path)]
|
||||||
|
for _source, path, title, summary in docs:
|
||||||
|
line = f"{path} — {title}"
|
||||||
|
first = _first_summary_line(summary)
|
||||||
|
if first:
|
||||||
|
line += f" — {first}"
|
||||||
|
lines.append(line)
|
||||||
|
user = "\n".join(lines)
|
||||||
|
limit = (
|
||||||
|
max_chars
|
||||||
|
if max_chars is not None
|
||||||
|
else get_settings().folder_summary_input_max_chars
|
||||||
|
)
|
||||||
|
if len(user) > limit:
|
||||||
|
user = user[:limit] + "\n" + TRUNCATION_MARKER
|
||||||
|
return SYSTEM_PROMPT, user
|
||||||
|
|
||||||
|
|
||||||
|
async def summarize_folder(
|
||||||
|
source: str, folder_path: str, docs: Sequence[DocRow], llm: FolderSummaryLLM
|
||||||
|
) -> str:
|
||||||
|
"""One-shot ``lite`` summary of one folder's recursive subtree.
|
||||||
|
|
||||||
|
Builds the prompt from the folder's documents (each ``path``,
|
||||||
|
``title``, first summary line), makes ONE :meth:`LLMClient.chat`
|
||||||
|
call against ``llm.settings.llm_summary_model`` (the ``lite``
|
||||||
|
model — no new model management), and returns the model's text
|
||||||
|
trimmed. Raises :class:`LLMError` when the reply is empty after
|
||||||
|
trimming (the client already rejects empty content; re-asserted
|
||||||
|
defensively, the summarizer's rule — a silent summary must never be
|
||||||
|
stored), and propagates any :class:`LLMError` the client raises
|
||||||
|
(the generator's fail-soft path catches it per folder).
|
||||||
|
"""
|
||||||
|
system, user = build_folder_summary_prompt(source, folder_path, docs)
|
||||||
|
raw = await llm.chat(
|
||||||
|
[{"role": "system", "content": system}, {"role": "user", "content": user}],
|
||||||
|
model=llm.settings.llm_summary_model,
|
||||||
|
)
|
||||||
|
summary = raw.strip()
|
||||||
|
if not summary:
|
||||||
|
label = _folder_label(source, folder_path)
|
||||||
|
raise LLMError(
|
||||||
|
f"folder summary model returned empty content for {label} — "
|
||||||
|
"refusing to store a silent summary"
|
||||||
|
)
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
def _upsert(db: Session, source: str, folder_path: str, summary: str) -> None:
|
||||||
|
"""Insert or update the row for one folder (fresh UTC stamp)."""
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
row = db.get(FolderSummary, (source, folder_path))
|
||||||
|
if row is None:
|
||||||
|
db.add(
|
||||||
|
FolderSummary(
|
||||||
|
source=source, folder_path=folder_path, summary=summary, updated_at=now
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
row.summary = summary
|
||||||
|
row.updated_at = now
|
||||||
|
|
||||||
|
|
||||||
|
def folder_summary_table_empty(db: Session) -> bool:
|
||||||
|
"""Whether ``folder_summaries`` holds no rows (the sync-path gate).
|
||||||
|
|
||||||
|
Phase 94 (task 02): the ``_overview_row_exists`` pattern
|
||||||
|
(``scripts.import_docs``) extended to a table-emptiness check —
|
||||||
|
after an unchanged re-sync, an EMPTY table (the first full sync
|
||||||
|
after migration 0017, or after a ``--limit`` debug walk that
|
||||||
|
skipped generation) still gets a fresh batch, while a populated
|
||||||
|
table is left untouched until the KB actually changes. One
|
||||||
|
bounded ``LIMIT 1`` probe, never a count scan.
|
||||||
|
"""
|
||||||
|
return db.execute(select(FolderSummary.source).limit(1)).first() is None
|
||||||
|
|
||||||
|
|
||||||
|
async def generate_folder_summaries(
|
||||||
|
db: Session, llm: FolderSummaryLLM, *, skip: bool = False
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""Regenerate the stored folder summaries for the current catalogue.
|
||||||
|
|
||||||
|
The sync-path orchestrator (phase 94, task 02 calls it
|
||||||
|
change-gated, like the KB overview; ``--limit`` debug runs pass
|
||||||
|
``skip=True``). Steps, in order:
|
||||||
|
|
||||||
|
1. ``skip=True`` → a no-op: the zero stats dict is returned, the
|
||||||
|
LLM is never called, and no rows are touched.
|
||||||
|
2. Group the document catalogue by :func:`group_by_folder` (the
|
||||||
|
recursive-subtree concept) and keep the candidate folders —
|
||||||
|
the ones whose recursive subtree holds
|
||||||
|
:data:`MIN_DOCS_PER_FOLDER` (≥ 2) documents. Single-document
|
||||||
|
folders get no row (their one file line IS their summary).
|
||||||
|
3. For each candidate (deterministic ``(source, folder_path)``
|
||||||
|
order) call :func:`summarize_folder` and UPSERT — fail-soft PER
|
||||||
|
FOLDER: one folder's :class:`LLMError` is logged and counted,
|
||||||
|
its previous row (if any) is kept, and the remaining folders
|
||||||
|
still land (a ``lite`` outage must never fail the sync — the KB
|
||||||
|
is the product, the summaries are auxiliary).
|
||||||
|
4. DELETE rows whose folder no longer has ≥ 2 documents — a
|
||||||
|
pruned/renamed folder's summary goes stale and is dropped.
|
||||||
|
Rows for folders that still qualify persist (regenerated in
|
||||||
|
step 3 — an unchanged folder's summary is still true).
|
||||||
|
|
||||||
|
Only flushes — the CALLER commits (the phase-53
|
||||||
|
``bump_sources_version`` convention: the sync path owns the
|
||||||
|
transaction, so a failed sync rolls the summaries back with it).
|
||||||
|
|
||||||
|
Returns the small stats dict ``{"generated", "failed", "pruned"}``
|
||||||
|
for the caller's summary-line logging (PLAN §9 ample logging).
|
||||||
|
"""
|
||||||
|
stats = {"generated": 0, "failed": 0, "pruned": 0}
|
||||||
|
if skip:
|
||||||
|
return stats
|
||||||
|
|
||||||
|
result = db.execute(
|
||||||
|
select(Document.source, Document.path, Document.title, Document.summary)
|
||||||
|
.order_by(Document.source, Document.path)
|
||||||
|
).all()
|
||||||
|
rows: list[DocRow] = [
|
||||||
|
(source, path, title, summary)
|
||||||
|
for source, path, title, summary in result
|
||||||
|
]
|
||||||
|
groups = group_by_folder(rows)
|
||||||
|
candidates = {
|
||||||
|
key: docs for key, docs in groups.items() if len(docs) >= MIN_DOCS_PER_FOLDER
|
||||||
|
}
|
||||||
|
|
||||||
|
for source, folder_path in sorted(candidates):
|
||||||
|
docs = candidates[(source, folder_path)]
|
||||||
|
try:
|
||||||
|
summary = await summarize_folder(source, folder_path, docs, llm)
|
||||||
|
except LLMError as e:
|
||||||
|
stats["failed"] += 1
|
||||||
|
logger.error(
|
||||||
|
"folder summary failed for %s/%s — %s", source, folder_path, e
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
_upsert(db, source, folder_path, summary)
|
||||||
|
stats["generated"] += 1
|
||||||
|
|
||||||
|
existing = db.execute(
|
||||||
|
select(FolderSummary.source, FolderSummary.folder_path)
|
||||||
|
).all()
|
||||||
|
for source, folder_path in existing:
|
||||||
|
if (source, folder_path) not in candidates:
|
||||||
|
row = db.get(FolderSummary, (source, folder_path))
|
||||||
|
if row is not None:
|
||||||
|
db.delete(row)
|
||||||
|
stats["pruned"] += 1
|
||||||
|
|
||||||
|
db.flush()
|
||||||
|
logger.info(
|
||||||
|
"folder_summaries: generated=%d failed=%d pruned=%d",
|
||||||
|
stats["generated"],
|
||||||
|
stats["failed"],
|
||||||
|
stats["pruned"],
|
||||||
|
)
|
||||||
|
return stats
|
||||||
+44
-19
@@ -31,7 +31,15 @@ name*, not a directory or file path, and ``read``/``grep`` take the
|
|||||||
combined ``source/path`` string *including the source name* (a bare
|
combined ``source/path`` string *including the source name* (a bare
|
||||||
document path will not resolve) — the same two things the phase-72
|
document path will not resolve) — the same two things the phase-72
|
||||||
teaching refusals in :mod:`app.rag.agent` re-state after the fact, so
|
teaching refusals in :mod:`app.rag.agent` re-state after the fact, so
|
||||||
the model carries the contract before it calls a tool): the **HIGH**
|
the model carries the contract before it calls a tool; phase 94,
|
||||||
|
task 03: the ``ls`` clause is rewritten to the drill-down tree
|
||||||
|
contract — no ``path``: the synced sources with counts + summaries, a
|
||||||
|
source name: its top-level folders + files, a ``source/folder`` path:
|
||||||
|
one level deeper — a listing shows only that level's subfolders + own
|
||||||
|
files (never the whole KB in one call), the folder summaries say
|
||||||
|
what's in a folder before drilling, and ``grep`` stays the locator for
|
||||||
|
finding one document without listing; the ``read``/``grep`` clauses
|
||||||
|
and the discipline rules are byte-identical): the **HIGH**
|
||||||
prompt only carries a ``<tools>`` section after the ``<documents>``
|
prompt only carries a ``<tools>`` section after the ``<documents>``
|
||||||
body — the grounded turn may extend its context through the three
|
body — the grounded turn may extend its context through the three
|
||||||
server-side tools (round-capped, see :mod:`app.rag.agent`; the cap is
|
server-side tools (round-capped, see :mod:`app.rag.agent`; the cap is
|
||||||
@@ -103,7 +111,17 @@ _KB_INTRO = (
|
|||||||
#: ``path`` is a *source name* (not a directory or file path) and
|
#: ``path`` is a *source name* (not a directory or file path) and
|
||||||
#: ``read``/``grep`` take the combined ``source/path`` string *including
|
#: ``read``/``grep`` take the combined ``source/path`` string *including
|
||||||
#: the source name* (a bare document path will not resolve) — the same
|
#: the source name* (a bare document path will not resolve) — the same
|
||||||
#: two things the phase-72 teaching refusals re-state after the fact):
|
#: two things the phase-72 teaching refusals re-state after the fact;
|
||||||
|
#: phase 94, task 03 — the owner-permitted tool-surface revision,
|
||||||
|
#: recorded in the phase 94 overview: the ``ls`` clause is rewritten
|
||||||
|
#: to the drill-down tree contract — no ``path``: the synced sources
|
||||||
|
#: with counts + summaries, a source name: its top-level folders +
|
||||||
|
#: files, a ``source/folder`` path: one level deeper — a listing shows
|
||||||
|
#: only that level's subfolders + own files, never the whole KB in one
|
||||||
|
#: call, the folder summaries say what's in a folder before drilling,
|
||||||
|
#: and ``grep`` stays the locator for finding one document without
|
||||||
|
#: listing; the ``read``/``grep`` clauses and the discipline rules are
|
||||||
|
#: byte-identical):
|
||||||
#: a grounded turn may extend its context through the three server-side
|
#: a grounded turn may extend its context through the three server-side
|
||||||
#: tools (round cap: ``BOR_AGENT_MAX_ROUNDS`` — the cap is the bound and
|
#: tools (round cap: ``BOR_AGENT_MAX_ROUNDS`` — the cap is the bound and
|
||||||
#: this section does not re-state it, phase 45). Appended after the mode
|
#: this section does not re-state it, phase 45). Appended after the mode
|
||||||
@@ -139,23 +157,30 @@ _KB_INTRO = (
|
|||||||
TOOLS_SECTION: str = (
|
TOOLS_SECTION: str = (
|
||||||
"<tools>\n"
|
"<tools>\n"
|
||||||
"You may extend your context with three tools. `ls` lists the "
|
"You may extend your context with three tools. `ls` lists the "
|
||||||
"indexed documents as `source: X | path: Y | title: Z` lines; its "
|
"knowledge base as a tree, one level at a time: with no `path` it "
|
||||||
"optional `path` argument is a source name (e.g. 'homelab'), not a "
|
"lists every synced source with its document count and a summary "
|
||||||
"directory or file path — omit it to list every document. `read` "
|
"of its contents; with a source name (e.g. 'homelab') it lists "
|
||||||
"pulls in one document by its combined `source/path` string, "
|
"that source's top-level folders and files; with a `source/folder` "
|
||||||
"exactly as shown in the `ls` output — including the source name — "
|
"path it drills one level deeper. A listing shows only that "
|
||||||
"adding its full content to your context. Do not call `read` for a "
|
"level's subfolders and its own files — never the whole knowledge "
|
||||||
"document already shown in the <documents> section, even when the "
|
"base in one call — and each folder line's summary says what the "
|
||||||
"user asks you to open or read it — its full text is already in "
|
"folder contains before you drill into it. File lines are "
|
||||||
"your prompt; answer directly from it. For `read`, a bare document "
|
"`source: X | path: Y | title: Z`; to find one specific document "
|
||||||
"path (without the source name) will not resolve. `grep` locates an "
|
"without listing, use `grep`. `read` pulls in one document by its "
|
||||||
"exact string (case-insensitive) in the indexed documents and "
|
"combined `source/path` string, exactly as shown in the `ls` "
|
||||||
"returns up to 20 matching `source/path:line: text` lines — a "
|
"output — including the source name — adding its full content to "
|
||||||
"locator, not a context-adder: read the winner with `read`. A grep "
|
"your context. Do not call `read` for a document already shown in "
|
||||||
"pattern is a plain substring, NEVER a regex — '.*' and '\\.' are "
|
"the <documents> section, even when the user asks you to open or "
|
||||||
"literal text there; if such a pattern returns no matches, retry "
|
"read it — its full text is already in your prompt; answer "
|
||||||
"with the plain text you expect to see. For a normal search pass "
|
"directly from it. For `read`, a bare document path (without the "
|
||||||
"only `pattern` — its optional `path` argument "
|
"source name) will not resolve. `grep` locates an exact string "
|
||||||
|
"(case-insensitive) in the indexed documents and returns up to 20 "
|
||||||
|
"matching `source/path:line: text` lines — a locator, not a "
|
||||||
|
"context-adder: read the winner with `read`. A grep pattern is a "
|
||||||
|
"plain substring, NEVER a regex — '.*' and '\\.' are literal text "
|
||||||
|
"there; if such a pattern returns no matches, retry with the plain "
|
||||||
|
"text you expect to see. For a normal search pass only `pattern` — "
|
||||||
|
"its optional `path` argument "
|
||||||
"limits the search to one document you already know, by the same "
|
"limits the search to one document you already know, by the same "
|
||||||
"combined `source/path` string; never a source name — a bare "
|
"combined `source/path` string; never a source name — a bare "
|
||||||
"document path (without the source name) will not resolve there "
|
"document path (without the source name) will not resolve there "
|
||||||
|
|||||||
@@ -82,17 +82,18 @@ from pathlib import Path
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
from app.api.chat import plan_turn
|
from app.api.chat import plan_turn
|
||||||
from app.api.steering import load_steering_notes
|
from app.api.steering import load_steering_notes
|
||||||
from app.config import Settings, get_settings
|
from app.config import Settings, get_settings
|
||||||
from app.db import SessionLocal, db_available
|
from app.db import SessionLocal, db_available
|
||||||
|
from app.models import Document
|
||||||
from app.rag.agent import (
|
from app.rag.agent import (
|
||||||
CORRECTION_INSTRUCTION,
|
CORRECTION_INSTRUCTION,
|
||||||
AgentHolder,
|
AgentHolder,
|
||||||
MalformedReplyError,
|
MalformedReplyError,
|
||||||
find_document,
|
find_document,
|
||||||
list_catalog,
|
|
||||||
list_source_names,
|
list_source_names,
|
||||||
run_agent,
|
run_agent,
|
||||||
)
|
)
|
||||||
@@ -113,6 +114,19 @@ logger = logging.getLogger("agent_realmodel_check")
|
|||||||
#: Repo-relative fixture dump (written by scripts.load_test_kb).
|
#: Repo-relative fixture dump (written by scripts.load_test_kb).
|
||||||
DEFAULT_DUMP_PATH = Path("tests/fixtures/test_kb.dump.sql")
|
DEFAULT_DUMP_PATH = Path("tests/fixtures/test_kb.dump.sql")
|
||||||
|
|
||||||
|
|
||||||
|
def _catalog(db) -> list[tuple[str, str, str]]:
|
||||||
|
"""The full catalog as ``(source, path, title)`` in ``(source, path)``
|
||||||
|
order — the battery builder's pre-phase-94 ``list_catalog`` contract,
|
||||||
|
kept local to this script (phase 94 deleted the agent accessor: the
|
||||||
|
drill-down ``ls`` lists one bounded source level at a time)."""
|
||||||
|
rows = db.execute(
|
||||||
|
select(Document.source, Document.path, Document.title).order_by(
|
||||||
|
Document.source, Document.path
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
return [(source, path, title) for source, path, title in rows]
|
||||||
|
|
||||||
#: The per-turn line truncates the question at this width (the locked
|
#: The per-turn line truncates the question at this width (the locked
|
||||||
#: format prints ``turn NN | emitted=E executed=X cap=Y|N | <question>``).
|
#: format prints ``turn NN | emitted=E executed=X cap=Y|N | <question>``).
|
||||||
QUESTION_DISPLAY_WIDTH = 40
|
QUESTION_DISPLAY_WIDTH = 40
|
||||||
@@ -263,7 +277,7 @@ def check_preconditions(
|
|||||||
return 2
|
return 2
|
||||||
return None
|
return None
|
||||||
with SessionLocal() as db:
|
with SessionLocal() as db:
|
||||||
catalog = list_catalog(db)
|
catalog = _catalog(db)
|
||||||
if len(catalog) < 2:
|
if len(catalog) < 2:
|
||||||
print(
|
print(
|
||||||
f"precondition failed: catalog holds {len(catalog)} document(s) "
|
f"precondition failed: catalog holds {len(catalog)} document(s) "
|
||||||
@@ -443,8 +457,33 @@ async def _run_deflected(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _ls_folder_prefixes(catalog: set[tuple[str, str]]) -> dict[str, set[str]]:
|
||||||
|
"""The existing ``ls`` folders per source (the phase-94 drill-down
|
||||||
|
contract): for each source, the set of source-relative folder
|
||||||
|
prefixes — every slash-boundary prefix of its indexed paths.
|
||||||
|
|
||||||
|
Mirrors the ``00_phase.md`` existence rule that
|
||||||
|
``app.rag.agent._execute_tool`` applies: a folder ``F`` exists ⟺
|
||||||
|
some indexed path of the source starts with ``F + "/"`` — a
|
||||||
|
document's OWN path is never a folder, so the document's path itself
|
||||||
|
is not in the set (``ls`` of a file path is a refusal, as is a bare
|
||||||
|
folder name missing its source prefix).
|
||||||
|
"""
|
||||||
|
folders: dict[str, set[str]] = {}
|
||||||
|
for source, path in catalog:
|
||||||
|
folder = path[: path.rfind("/")] if "/" in path else ""
|
||||||
|
while folder:
|
||||||
|
folders.setdefault(source, set()).add(folder)
|
||||||
|
folder = folder[: folder.rfind("/")] if "/" in folder else ""
|
||||||
|
return folders
|
||||||
|
|
||||||
|
|
||||||
def classify_call(
|
def classify_call(
|
||||||
name: str, args: dict[str, Any], catalog: set[tuple[str, str]], sources: set[str]
|
name: str,
|
||||||
|
args: dict[str, Any],
|
||||||
|
catalog: set[tuple[str, str]],
|
||||||
|
sources: set[str],
|
||||||
|
ls_folders: dict[str, set[str]],
|
||||||
) -> bool:
|
) -> bool:
|
||||||
"""Contract correctness of ONE emitted call (the tool-calling
|
"""Contract correctness of ONE emitted call (the tool-calling
|
||||||
accuracy metric, 2026-09-04 controlled methodology).
|
accuracy metric, 2026-09-04 controlled methodology).
|
||||||
@@ -460,12 +499,25 @@ def classify_call(
|
|||||||
re-read 15/15 across copy variants — is documented in
|
re-read 15/15 across copy variants — is documented in
|
||||||
``TOOL_CALLING_TESTING.md``). The classification mirrors
|
``TOOL_CALLING_TESTING.md``). The classification mirrors
|
||||||
``app.rag.agent._execute_tool``'s resolution rules gate-side (no
|
``app.rag.agent._execute_tool``'s resolution rules gate-side (no
|
||||||
app-code changes for measurement).
|
app-code changes for measurement) — including the phase-94 ``ls``
|
||||||
|
revision (owner-permitted tool-surface change, ``00_phase.md``):
|
||||||
|
``ls(path)`` is contract-correct for ``""`` (the synced sources),
|
||||||
|
a registered source name (its root folder), or a ``source/folder``
|
||||||
|
path naming an EXISTING folder (``ls_folders`` — the drill-down);
|
||||||
|
an unknown first segment, a bare folder name (no source prefix), or
|
||||||
|
a folder matching no indexed prefix is the violation.
|
||||||
"""
|
"""
|
||||||
if name == "ls":
|
if name == "ls":
|
||||||
raw = args.get("path")
|
raw = args.get("path")
|
||||||
scope = raw.strip() if isinstance(raw, str) else ""
|
scope = raw.strip() if isinstance(raw, str) else ""
|
||||||
return scope == "" or scope in sources
|
if not scope:
|
||||||
|
return True # the top level (the synced sources)
|
||||||
|
source, _, rest = scope.partition("/")
|
||||||
|
if source not in sources:
|
||||||
|
return False # unknown first segment (the phase-72 incident class)
|
||||||
|
if not rest:
|
||||||
|
return True # a registered source's root folder
|
||||||
|
return rest in ls_folders.get(source, ())
|
||||||
if name == "read":
|
if name == "read":
|
||||||
raw = args.get("path")
|
raw = args.get("path")
|
||||||
arg = raw.strip() if isinstance(raw, str) else ""
|
arg = raw.strip() if isinstance(raw, str) else ""
|
||||||
@@ -498,10 +550,13 @@ def score_contract(
|
|||||||
and source names. Per-turn counts are attached on each
|
and source names. Per-turn counts are attached on each
|
||||||
:class:`TurnResult` as ``_contract_ok`` (measurement state, not a
|
:class:`TurnResult` as ``_contract_ok`` (measurement state, not a
|
||||||
dataclass field — the display line stays the locked format)."""
|
dataclass field — the display line stays the locked format)."""
|
||||||
|
ls_folders = _ls_folder_prefixes(catalog)
|
||||||
ok = 0
|
ok = 0
|
||||||
for turn in turns:
|
for turn in turns:
|
||||||
turn_ok = sum(
|
turn_ok = sum(
|
||||||
1 for name, args in turn.calls if classify_call(name, args, catalog, sources)
|
1
|
||||||
|
for name, args in turn.calls
|
||||||
|
if classify_call(name, args, catalog, sources, ls_folders)
|
||||||
)
|
)
|
||||||
turn._contract_ok = turn_ok # type: ignore[attr-defined]
|
turn._contract_ok = turn_ok # type: ignore[attr-defined]
|
||||||
ok += turn_ok
|
ok += turn_ok
|
||||||
@@ -763,7 +818,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
with SessionLocal() as db:
|
with SessionLocal() as db:
|
||||||
catalog = list_catalog(db)
|
catalog = _catalog(db)
|
||||||
s2, p2, _t2 = catalog[1]
|
s2, p2, _t2 = catalog[1]
|
||||||
d2 = find_document(db, s2, p2)
|
d2 = find_document(db, s2, p2)
|
||||||
d2_content = d2.content if d2 is not None else ""
|
d2_content = d2.content if d2 is not None else ""
|
||||||
@@ -794,7 +849,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
# source names (the KB is static across the run — the restore, when
|
# source names (the KB is static across the run — the restore, when
|
||||||
# any, happened before the battery).
|
# any, happened before the battery).
|
||||||
with SessionLocal() as db:
|
with SessionLocal() as db:
|
||||||
catalog_set = set((s, p) for s, p, _t in list_catalog(db))
|
catalog_set = set((s, p) for s, p, _t in _catalog(db))
|
||||||
sources_set = set(list_source_names(db))
|
sources_set = set(list_source_names(db))
|
||||||
score_contract(turns, catalog_set, sources_set)
|
score_contract(turns, catalog_set, sources_set)
|
||||||
passed, conditions = evaluate(
|
passed, conditions = evaluate(
|
||||||
|
|||||||
+93
-15
@@ -54,6 +54,20 @@ reports ``overview=failed`` on the summary line: the import's exit code
|
|||||||
is about files, and the previous outline stays (an old outline is better
|
is about files, and the previous outline stays (an old outline is better
|
||||||
than none).
|
than none).
|
||||||
|
|
||||||
|
The stored folder summaries (phase 94 — the drill-down ``ls``'s
|
||||||
|
per-level descriptions, the ``folder_summaries`` table) regenerate in
|
||||||
|
the same run under the same gate: a changed KB (added + updated > 0),
|
||||||
|
or an empty table after an unchanged walk (the first full run after
|
||||||
|
migration 0017, or after a ``--limit`` first walk that skipped them).
|
||||||
|
Same contract — **best-effort, per-folder fail-soft**: a ``lite``
|
||||||
|
failure keeps the failed folders' previous rows and only counts into
|
||||||
|
the stats; ``--limit`` debug runs skip them entirely (never burning a
|
||||||
|
``lite`` call). The stats land on the summary line as
|
||||||
|
``folder_summaries=<generated>/<failed>/<pruned>`` (or
|
||||||
|
``folder_summaries=skipped`` when the gate did not fire), and the rows
|
||||||
|
commit in the run's own short-lived session (the phase-53 convention —
|
||||||
|
a failed commit rolls them back with it).
|
||||||
|
|
||||||
A run that **changed** the knowledge base (added + updated + pruned > 0
|
A run that **changed** the knowledge base (added + updated + pruned > 0
|
||||||
— phase 53, task 02) also advances the single-row ``sources_meta``
|
— phase 53, task 02) also advances the single-row ``sources_meta``
|
||||||
version exactly once (``sources_version=<n>`` on the summary line): the
|
version exactly once (``sources_version=<n>`` on the summary line): the
|
||||||
@@ -78,6 +92,7 @@ from app.core.debugging import configure_debugging
|
|||||||
from app.core.logging import configure_logging
|
from app.core.logging import configure_logging
|
||||||
from app.db import SessionLocal
|
from app.db import SessionLocal
|
||||||
from app.models import KbOverview
|
from app.models import KbOverview
|
||||||
|
from app.rag.folder_summaries import folder_summary_table_empty, generate_folder_summaries
|
||||||
from app.rag.git_sources import effective_sources
|
from app.rag.git_sources import effective_sources
|
||||||
from app.rag.importer import ImportSummary, import_sources
|
from app.rag.importer import ImportSummary, import_sources
|
||||||
from app.rag.llm import LLMClient
|
from app.rag.llm import LLMClient
|
||||||
@@ -216,6 +231,20 @@ def _overview_row_exists() -> bool:
|
|||||||
return session.get(KbOverview, 1) is not None
|
return session.get(KbOverview, 1) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def _folder_summaries_table_empty() -> bool:
|
||||||
|
"""Whether the ``folder_summaries`` table holds any row (phase 94).
|
||||||
|
|
||||||
|
The ``_overview_row_exists`` pattern extended to a table-emptiness
|
||||||
|
check (one bounded ``LIMIT 1`` probe): an empty table after an
|
||||||
|
unchanged re-import — e.g. the first full run after migration 0017,
|
||||||
|
or after a ``--limit`` first walk that skipped generation — still
|
||||||
|
gets a fresh batch of folder summaries, while a populated table is
|
||||||
|
left untouched until the KB actually changes.
|
||||||
|
"""
|
||||||
|
with SessionLocal() as session:
|
||||||
|
return folder_summary_table_empty(session)
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
def main(argv: list[str] | None = None) -> int:
|
||||||
args = build_parser().parse_args(argv)
|
args = build_parser().parse_args(argv)
|
||||||
settings = get_settings()
|
settings = get_settings()
|
||||||
@@ -246,9 +275,10 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
|
|
||||||
llm = LLMClient()
|
llm = LLMClient()
|
||||||
|
|
||||||
async def _run() -> tuple[ImportSummary, str, str]:
|
async def _run() -> tuple[ImportSummary, str, str, dict[str, int] | None]:
|
||||||
"""Import, then (change-gated) advance the sources version and
|
"""Import, then (change-gated) advance the sources version,
|
||||||
refresh the stored KB overview.
|
refresh the stored KB overview, and regenerate the stored folder
|
||||||
|
summaries.
|
||||||
|
|
||||||
One event loop, one ``LLMClient`` (phase 31, task 04): the
|
One event loop, one ``LLMClient`` (phase 31, task 04): the
|
||||||
outline that every chat prompt injects as ``<knowledge_base>`` is
|
outline that every chat prompt injects as ``<knowledge_base>`` is
|
||||||
@@ -268,6 +298,17 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
deliberately broader than the overview's). ``--limit`` debug
|
deliberately broader than the overview's). ``--limit`` debug
|
||||||
runs and unchanged re-runs never bump. The returned token is
|
runs and unchanged re-runs never bump. The returned token is
|
||||||
the new version, or ``"skipped"``.
|
the new version, or ``"skipped"``.
|
||||||
|
|
||||||
|
The folder summaries (phase 94, task 02) follow the same gate
|
||||||
|
— a changed KB, or an empty ``folder_summaries`` table after an
|
||||||
|
unchanged walk (the first full run after migration 0017, or
|
||||||
|
after a ``--limit`` first walk) — with per-folder fail-soft
|
||||||
|
inside the generator (a ``lite`` failure keeps the failed
|
||||||
|
folders' previous rows). The generator only flushes: this
|
||||||
|
run's own short-lived session commits (the phase-53
|
||||||
|
convention), and the stats land on the summary line as
|
||||||
|
``folder_summaries=<generated>/<failed>/<pruned>``
|
||||||
|
(``None`` — rendered ``skipped`` — when the gate did not fire).
|
||||||
"""
|
"""
|
||||||
summary = await import_sources(
|
summary = await import_sources(
|
||||||
sources, llm, prune=args.prune, limit=args.limit,
|
sources, llm, prune=args.prune, limit=args.limit,
|
||||||
@@ -294,32 +335,69 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
sources_version = "skipped"
|
sources_version = "skipped"
|
||||||
logger.info("sources: version bump skipped (KB unchanged)")
|
logger.info("sources: version bump skipped (KB unchanged)")
|
||||||
if args.limit is not None:
|
if args.limit is not None:
|
||||||
|
# An incomplete walk must never rewrite the outline or the
|
||||||
|
# folder summaries (the --limit skip, mirrored above for the
|
||||||
|
# sources version).
|
||||||
logger.info("overview: skipped (--limit)")
|
logger.info("overview: skipped (--limit)")
|
||||||
return summary, "skipped", sources_version
|
return summary, "skipped", sources_version, None
|
||||||
if summary.added + summary.updated == 0:
|
changed = summary.added + summary.updated > 0
|
||||||
|
overview_due = changed
|
||||||
|
folders_due = changed
|
||||||
|
if not changed:
|
||||||
if summary.files == 0:
|
if summary.files == 0:
|
||||||
logger.info("overview: skipped (nothing imported)")
|
logger.info("overview: skipped (nothing imported)")
|
||||||
return summary, "skipped", sources_version
|
return summary, "skipped", sources_version, None
|
||||||
if _overview_row_exists():
|
# The unchanged-walk first-run triggers: the overview when
|
||||||
|
# no row exists yet (the first run after migration 0005),
|
||||||
|
# the folder summaries when the table is empty (the first
|
||||||
|
# full run after migration 0017, or after a --limit first
|
||||||
|
# walk that skipped them).
|
||||||
|
overview_due = not _overview_row_exists()
|
||||||
|
folders_due = _folder_summaries_table_empty()
|
||||||
|
if not overview_due and not folders_due:
|
||||||
logger.info("overview: skipped (KB unchanged)")
|
logger.info("overview: skipped (KB unchanged)")
|
||||||
return summary, "skipped", sources_version
|
return summary, "skipped", sources_version, None
|
||||||
# No outline yet after an unchanged re-import (e.g. the first
|
overview_status = "skipped"
|
||||||
# run after migration 0005) — fall through and generate one.
|
if overview_due:
|
||||||
ok = await regenerate_overview(llm)
|
ok = await regenerate_overview(llm)
|
||||||
return summary, "updated" if ok else "failed", sources_version
|
overview_status = "updated" if ok else "failed"
|
||||||
|
else:
|
||||||
|
logger.info("overview: skipped (KB unchanged)")
|
||||||
|
# Phase 94 (task 02): the folder summaries — same event loop +
|
||||||
|
# LLMClient (phase 31 convention), per-folder fail-soft inside
|
||||||
|
# the generator (a ``lite`` failure never flips the exit code).
|
||||||
|
# It only flushes — this run's own short-lived session commits
|
||||||
|
# (the phase-53 convention), so a failed commit rolls the
|
||||||
|
# summaries back with it.
|
||||||
|
folder_stats: dict[str, int] | None = None
|
||||||
|
if folders_due:
|
||||||
|
session = SessionLocal()
|
||||||
|
try:
|
||||||
|
folder_stats = await generate_folder_summaries(session, llm)
|
||||||
|
session.commit()
|
||||||
|
finally:
|
||||||
|
session.close()
|
||||||
|
return summary, overview_status, sources_version, folder_stats
|
||||||
|
|
||||||
summary, overview_status, sources_version = asyncio.run(_run())
|
summary, overview_status, sources_version, folder_stats = asyncio.run(_run())
|
||||||
|
folder_token = (
|
||||||
|
"skipped"
|
||||||
|
if folder_stats is None
|
||||||
|
else f"{folder_stats['generated']}/{folder_stats['failed']}/{folder_stats['pruned']}"
|
||||||
|
)
|
||||||
print(
|
print(
|
||||||
f"import_docs: files={summary.files} added={summary.added} "
|
f"import_docs: files={summary.files} added={summary.added} "
|
||||||
f"updated={summary.updated} unchanged={summary.unchanged} "
|
f"updated={summary.updated} unchanged={summary.unchanged} "
|
||||||
f"pruned={summary.pruned} errors={summary.errors} chunks={summary.chunks} "
|
f"pruned={summary.pruned} errors={summary.errors} chunks={summary.chunks} "
|
||||||
f"embed_batches={summary.embed_batches} summaries={summary.summaries} "
|
f"embed_batches={summary.embed_batches} summaries={summary.summaries} "
|
||||||
f"summary_errors={summary.summary_errors} formats={summary.format_counts()} "
|
f"summary_errors={summary.summary_errors} formats={summary.format_counts()} "
|
||||||
f"overview={overview_status} sources_version={sources_version}"
|
f"overview={overview_status} sources_version={sources_version} "
|
||||||
|
f"folder_summaries={folder_token}"
|
||||||
)
|
)
|
||||||
# Non-zero if any file failed, so cron/CI notice — the rest of the KB
|
# Non-zero if any file failed, so cron/CI notice — the rest of the KB
|
||||||
# was imported and the failed files are retried on the next run. The
|
# was imported and the failed files are retried on the next run. The
|
||||||
# overview is best-effort: a failed outline never changes the exit code.
|
# overview and the folder summaries are best-effort: a failed outline
|
||||||
|
# or a failed folder batch never changes the exit code.
|
||||||
return 1 if summary.errors else 0
|
return 1 if summary.errors else 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+51
-11
@@ -15,23 +15,30 @@ guessable by any model. This script:
|
|||||||
``BOR_GIT_SOURCES`` env var),
|
``BOR_GIT_SOURCES`` env var),
|
||||||
3. imports the fixture documents through the real pipeline
|
3. imports the fixture documents through the real pipeline
|
||||||
(``import_sources`` — real chunking + real ``embed``-model
|
(``import_sources`` — real chunking + real ``embed``-model
|
||||||
embeddings; this is the ONLY step that burns model calls, and only
|
embeddings; this is the only step that burns ``embed`` calls, and
|
||||||
at build time),
|
only at build time),
|
||||||
4. stores the static KB overview + the sources-version row,
|
4. stores the static KB overview + the sources-version row,
|
||||||
5. prints a **retrieval report** for every fixture-battery question
|
5. generates the sync-time **folder summaries** through the real
|
||||||
|
generator (``app.rag.folder_summaries``) against the live ``lite``
|
||||||
|
endpoint — the drill-down ``ls`` (phase 94) shows the stored rows,
|
||||||
|
so the dump must carry them (the build always truncates first, so
|
||||||
|
the sync paths' table-empty first-run trigger holds; a failed batch
|
||||||
|
aborts the build — a summary-less dump would be a broken fixture),
|
||||||
|
6. prints a **retrieval report** for every fixture-battery question
|
||||||
(grounded or deflected, which documents would seed) — the battery
|
(grounded or deflected, which documents would seed) — the battery
|
||||||
must be all-grounded for the gate to exercise the tools,
|
must be all-grounded for the gate to exercise the tools,
|
||||||
6. snapshots the resulting database state into
|
7. snapshots the resulting database state into
|
||||||
``tests/fixtures/test_kb.dump.sql`` — a data-only SQL script
|
``tests/fixtures/test_kb.dump.sql`` — a data-only SQL script
|
||||||
(TRUNCATE + one multi-row ``INSERT`` per app table, generated
|
(TRUNCATE + one multi-row ``INSERT`` per app table, generated
|
||||||
in-process — the same file runs in psql or psycopg, in one
|
in-process — the same file runs in psql or psycopg, in one
|
||||||
transaction) — and **verifies the snapshot by restoring it and
|
transaction) — and **verifies the snapshot by restoring it and
|
||||||
comparing a per-table checksum**.
|
comparing a per-table checksum**.
|
||||||
|
|
||||||
Re-run it only when the fixture documents, the chunker, or the
|
Re-run it only when the fixture documents, the chunker, the embedding
|
||||||
embedding model change — everyday iterations restore the dump in
|
model, or the folder-summary prompt (its ``lite`` output is baked into
|
||||||
sub-second time (``scripts/restore_test_kb`` / the gate's
|
the dump) change — everyday iterations restore the dump in sub-second
|
||||||
``--restore``), never re-embedding (see ``TOOL_CALLING_TESTING.md``).
|
time (``scripts/restore_test_kb`` / the gate's ``--restore``), never
|
||||||
|
re-embedding (see ``TOOL_CALLING_TESTING.md``).
|
||||||
|
|
||||||
Exit codes: **0** built + verified, **1** build/verification failure,
|
Exit codes: **0** built + verified, **1** build/verification failure,
|
||||||
**2** precondition failure.
|
**2** precondition failure.
|
||||||
@@ -59,6 +66,7 @@ from app.models import (
|
|||||||
Chunk,
|
Chunk,
|
||||||
DocDraft,
|
DocDraft,
|
||||||
Document,
|
Document,
|
||||||
|
FolderSummary,
|
||||||
GitSource,
|
GitSource,
|
||||||
KbOverview,
|
KbOverview,
|
||||||
QueryLog,
|
QueryLog,
|
||||||
@@ -66,6 +74,7 @@ from app.models import (
|
|||||||
SourcesMeta,
|
SourcesMeta,
|
||||||
SteeringNote,
|
SteeringNote,
|
||||||
)
|
)
|
||||||
|
from app.rag.folder_summaries import generate_folder_summaries
|
||||||
from app.rag.importer import import_sources
|
from app.rag.importer import import_sources
|
||||||
from app.rag.llm import LLMClient
|
from app.rag.llm import LLMClient
|
||||||
from app.rag.retriever import retrieve
|
from app.rag.retriever import retrieve
|
||||||
@@ -111,6 +120,12 @@ _TABLES: tuple[tuple[str, type, tuple[str, ...]], ...] = (
|
|||||||
("doc_drafts", DocDraft, ("id", "token", "title", "path", "body",
|
("doc_drafts", DocDraft, ("id", "token", "title", "path", "body",
|
||||||
"status", "branch", "commit_sha", "created_at",
|
"status", "branch", "commit_sha", "created_at",
|
||||||
"updated_at")),
|
"updated_at")),
|
||||||
|
# Phase 94 (task 05): the sync-time folder summaries the drill-down
|
||||||
|
# ``ls`` shows — generated against the live ``lite`` endpoint in
|
||||||
|
# step 5 of :func:`_build` (the build truncates first, so the sync
|
||||||
|
# paths' table-empty first-run trigger holds).
|
||||||
|
("folder_summaries", FolderSummary, ("source", "folder_path",
|
||||||
|
"summary", "updated_at")),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -257,7 +272,31 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
|
|||||||
db.add(meta)
|
db.add(meta)
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
# 4. Retrieval report (all battery questions must stay grounded).
|
# 4. The sync-time folder summaries (phase 94, task 05): the
|
||||||
|
# drill-down ``ls`` shows the stored rows, so the dump carries
|
||||||
|
# them. Generated against the live ``lite`` endpoint through the
|
||||||
|
# real sync-time generator — the build just truncated the table,
|
||||||
|
# so the sync paths' table-empty first-run trigger holds (no
|
||||||
|
# change-gate bookkeeping needed on a fresh build). The generator
|
||||||
|
# only flushes; this build commits (the phase-53 convention the
|
||||||
|
# sync paths follow). A failed batch aborts: the dump's folder
|
||||||
|
# lines are part of the controlled fixture.
|
||||||
|
with SessionLocal() as db:
|
||||||
|
folder_stats = await generate_folder_summaries(db, llm)
|
||||||
|
db.commit()
|
||||||
|
logger.info(
|
||||||
|
"load_test_kb: folder summaries generated=%d failed=%d pruned=%d",
|
||||||
|
folder_stats["generated"], folder_stats["failed"], folder_stats["pruned"],
|
||||||
|
)
|
||||||
|
if folder_stats["failed"]:
|
||||||
|
print(
|
||||||
|
f"load_test_kb: {folder_stats['failed']} folder summary call(s) "
|
||||||
|
"failed — the fixture dump needs the stored rows; check the LLM "
|
||||||
|
"endpoint (BOR_LLM_SUMMARY_MODEL) and re-run"
|
||||||
|
)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
# 5. Retrieval report (all battery questions must stay grounded).
|
||||||
n_deflected = await _retrieval_report(llm, list(FIXTURE_BATTERY))
|
n_deflected = await _retrieval_report(llm, list(FIXTURE_BATTERY))
|
||||||
if n_deflected:
|
if n_deflected:
|
||||||
print(
|
print(
|
||||||
@@ -267,7 +306,7 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
|
|||||||
"question before running the gate."
|
"question before running the gate."
|
||||||
)
|
)
|
||||||
|
|
||||||
# 5. Snapshot (data-only) + round-trip verification.
|
# 6. Snapshot (data-only) + round-trip verification.
|
||||||
with SessionLocal() as db:
|
with SessionLocal() as db:
|
||||||
before = {table: _table_checksum(db, table) for table, _m, _c in _TABLES}
|
before = {table: _table_checksum(db, table) for table, _m, _c in _TABLES}
|
||||||
parts = [
|
parts = [
|
||||||
@@ -309,7 +348,8 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
|
|||||||
wall = time.monotonic() - started
|
wall = time.monotonic() - started
|
||||||
print(
|
print(
|
||||||
f"load_test_kb: ok — docs={summary.added} chunks={summary.chunks} "
|
f"load_test_kb: ok — docs={summary.added} chunks={summary.chunks} "
|
||||||
f"sources={len(FIXTURE_SOURCES)} dump={dump_path} "
|
f"sources={len(FIXTURE_SOURCES)} folder_summaries={folder_stats['generated']} "
|
||||||
|
f"dump={dump_path} "
|
||||||
f"({dump_path.stat().st_size // 1024} KB, verified by round-trip) "
|
f"({dump_path.stat().st_size // 1024} KB, verified by round-trip) "
|
||||||
f"in {wall:.1f}s"
|
f"in {wall:.1f}s"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -11,11 +11,13 @@ snapshot in **one transaction** through the app's own database URL
|
|||||||
(``BOR_DATABASE_URL``): no git clone of the homelab repo, no re-embedding,
|
(``BOR_DATABASE_URL``): no git clone of the homelab repo, no re-embedding,
|
||||||
no ``lite``-model calls — the whole known state (documents, chunks +
|
no ``lite``-model calls — the whole known state (documents, chunks +
|
||||||
embeddings, the source registry rows, the KB overview, the sources
|
embeddings, the source registry rows, the KB overview, the sources
|
||||||
version) lands in a fraction of a second, which is what makes a
|
version, the stored folder summaries — the drill-down ``ls``'s rows,
|
||||||
tool-calling iteration loop fast (see ``TOOL_CALLING_TESTING.md``):
|
phase 94, task 05) lands in a fraction of a second, which is what makes
|
||||||
|
a tool-calling iteration loop fast (see ``TOOL_CALLING_TESTING.md``):
|
||||||
|
|
||||||
uv run python -m scripts.restore_test_kb
|
uv run python -m scripts.restore_test_kb
|
||||||
# restore_test_kb: ok in 0.41s (8 docs, 2 sources, 16 chunks)
|
# restore_test_kb: ok in 0.41s (8 docs, 2 sources, 9 chunks,
|
||||||
|
# 4 folder summaries, dump … KB)
|
||||||
|
|
||||||
The gate runs the same restore inline:
|
The gate runs the same restore inline:
|
||||||
``uv run python -m scripts.agent_realmodel_check --restore``.
|
``uv run python -m scripts.agent_realmodel_check --restore``.
|
||||||
@@ -57,6 +59,10 @@ APP_TABLES: tuple[str, ...] = (
|
|||||||
"query_log",
|
"query_log",
|
||||||
"saved_chats",
|
"saved_chats",
|
||||||
"doc_drafts",
|
"doc_drafts",
|
||||||
|
# Phase 94 (task 05): the sync-time folder summaries the drill-down
|
||||||
|
# ``ls`` shows — the dump carries the rows (built by
|
||||||
|
# ``scripts.load_test_kb`` against the live ``lite`` endpoint).
|
||||||
|
"folder_summaries",
|
||||||
)
|
)
|
||||||
|
|
||||||
#: Repo-relative default dump location (the load script writes it there).
|
#: Repo-relative default dump location (the load script writes it there).
|
||||||
@@ -71,6 +77,7 @@ class RestoreResult:
|
|||||||
docs: int
|
docs: int
|
||||||
sources: tuple[str, ...]
|
sources: tuple[str, ...]
|
||||||
chunks: int
|
chunks: int
|
||||||
|
folders: int # stored folder summaries (phase 94)
|
||||||
dump_bytes: int
|
dump_bytes: int
|
||||||
|
|
||||||
|
|
||||||
@@ -127,11 +134,13 @@ def restore_dump(dump: Path) -> RestoreResult:
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
chunks = db.execute(text("select count(*) from chunks")).scalar_one()
|
chunks = db.execute(text("select count(*) from chunks")).scalar_one()
|
||||||
|
folders = db.execute(text("select count(*) from folder_summaries")).scalar_one()
|
||||||
return RestoreResult(
|
return RestoreResult(
|
||||||
seconds=seconds,
|
seconds=seconds,
|
||||||
docs=docs,
|
docs=docs,
|
||||||
sources=sources,
|
sources=sources,
|
||||||
chunks=chunks,
|
chunks=chunks,
|
||||||
|
folders=folders,
|
||||||
dump_bytes=dump.stat().st_size,
|
dump_bytes=dump.stat().st_size,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -169,7 +178,8 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
print(
|
print(
|
||||||
f"restore_test_kb: ok in {result.seconds:.2f}s "
|
f"restore_test_kb: ok in {result.seconds:.2f}s "
|
||||||
f"({result.docs} docs, {len(result.sources)} sources, "
|
f"({result.docs} docs, {len(result.sources)} sources, "
|
||||||
f"{result.chunks} chunks, dump {result.dump_bytes // 1024} KB)"
|
f"{result.chunks} chunks, {result.folders} folder summaries, "
|
||||||
|
f"dump {result.dump_bytes // 1024} KB)"
|
||||||
)
|
)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|||||||
+456
-73
@@ -18,6 +18,14 @@ Implements just enough of the aipi surface:
|
|||||||
- ``KB_OVERVIEW_MODE`` -> the deterministic outline: the first 8 tokens
|
- ``KB_OVERVIEW_MODE`` -> the deterministic outline: the first 8 tokens
|
||||||
of the user message (the generator puts the document list there) —
|
of the user message (the generator puts the document list there) —
|
||||||
byte-stable for a given KB (KB overview, phase 31)
|
byte-stable for a given KB (KB overview, phase 31)
|
||||||
|
- ``FOLDER_SUMMARY_MODE`` -> the deterministic folder one-liner
|
||||||
|
``Fixture folder summary for <folder>.`` — <folder> is the user
|
||||||
|
message's ``Folder: …`` header line (the generator names the
|
||||||
|
source/folder there, phase 94), so the stored row always names its
|
||||||
|
folder and an E2E can assert on it. Checked BEFORE the
|
||||||
|
``SUMMARY_MODE`` branch: the folder marker CONTAINS the summary
|
||||||
|
marker as a substring, so the summary branch would otherwise
|
||||||
|
shadow every folder-summary call
|
||||||
- ``DEFLECT_MODE`` -> honest "I haven't done anything like that" answer
|
- ``DEFLECT_MODE`` -> honest "I haven't done anything like that" answer
|
||||||
- otherwise -> upbeat answer quoting the provided document context
|
- otherwise -> upbeat answer quoting the provided document context
|
||||||
- user message containing ``pretend to think slowly`` -> 3s warm-up delay
|
- user message containing ``pretend to think slowly`` -> 3s warm-up delay
|
||||||
@@ -57,22 +65,31 @@ Implements just enough of the aipi surface:
|
|||||||
closing tag — same sentinel semantics.)
|
closing tag — same sentinel semantics.)
|
||||||
- user message containing ``use your tools`` (phase 37, agent document
|
- user message containing ``use your tools`` (phase 37, agent document
|
||||||
tools; phase 70: the flow emits the harness-aligned names — ``ls``
|
tools; phase 70: the flow emits the harness-aligned names — ``ls``
|
||||||
/ ``read`` with the combined ``source/path`` identity) **and** the
|
/ ``read`` with the combined ``source/path`` identity; phase 94:
|
||||||
system prompt carries the ``<tools>`` section -> the deterministic
|
the flow DRILLS through the tree ``ls`` — the top-level listing
|
||||||
SINGLE-READ tool flow, discriminated statelessly from the messages
|
carries sources only, so the flow takes one drill step, ``ls``
|
||||||
(the ``tools`` parameter gates the list/read steps — a no-tools
|
scoped to the first source, before the first file line exists)
|
||||||
request with no tool results is not the flow):
|
**and** the system prompt carries the ``<tools>`` section -> the
|
||||||
|
deterministic SINGLE-READ tool flow, discriminated statelessly from
|
||||||
|
the messages (the ``tools`` parameter gates the list/read steps —
|
||||||
|
a no-tools request with no tool results is not the flow):
|
||||||
* request 1 (``tools`` offered, no tool results yet): stream ONLY
|
* request 1 (``tools`` offered, no tool results yet): stream ONLY
|
||||||
``tool_calls`` deltas — ``ls`` (synthetic id ``call_0``, no
|
``tool_calls`` deltas — ``ls`` (synthetic id ``call_0``, no
|
||||||
arguments), ``finish_reason: "tool_calls"``, no content;
|
arguments), ``finish_reason: "tool_calls"``, no content;
|
||||||
* request 2 (a ``tool``-role catalog result in the messages):
|
* request 2 (the top-level source listing in the messages — the
|
||||||
parse the FIRST catalog line (``source: X | path: Y | title: Z``
|
agent's ``ls`` tree format, phase 94: no file lines yet):
|
||||||
— the labeled ``source:`` / ``path:`` fields, phase 63) and
|
stream a ``tool_calls`` delta — ``ls`` scoped to the FIRST
|
||||||
stream a ``tool_calls`` delta calling ``read`` on the JOINED
|
source of the listing (id ``call_1``) — the drill step (one
|
||||||
combined ``source/path`` (the mock joins the two labeled fields
|
drill per flow — ``_drill_target`` skips already-drilled
|
||||||
— the catalog format is unchanged, so this join is the only
|
sources, so the second listing is the folder level);
|
||||||
parse change, phase 70) (id ``call_1``);
|
* request 3 (a ``tool``-role folder listing with file lines in
|
||||||
* request 3 (a ``tool``-role read result in the messages —
|
the messages): parse the FIRST file line (``source: X | path:
|
||||||
|
Y | title: Z`` — the labeled ``source:`` / ``path:`` fields,
|
||||||
|
phase 63) and stream a ``tool_calls`` delta calling ``read`` on
|
||||||
|
the JOINED combined ``source/path`` (the mock joins the two
|
||||||
|
labeled fields — the file-line format is unchanged, so this
|
||||||
|
join is the only parse change, phase 70) (id ``call_2``);
|
||||||
|
* request 4 (a ``tool``-role read result in the messages —
|
||||||
content starting with the agent's ``"Document <source/path>:"``
|
content starting with the agent's ``"Document <source/path>:"``
|
||||||
header): a content answer, deterministic: ``Read
|
header): a content answer, deterministic: ``Read
|
||||||
<source/path>. <first 80 chars of the read document's
|
<source/path>. <first 80 chars of the read document's
|
||||||
@@ -85,19 +102,22 @@ Implements just enough of the aipi surface:
|
|||||||
- user message containing BOTH ``use your tools`` AND ``read two
|
- user message containing BOTH ``use your tools`` AND ``read two
|
||||||
documents`` (``MULTI_READ_TRIGGER``, phase 45 task 02) **and** the
|
documents`` (``MULTI_READ_TRIGGER``, phase 45 task 02) **and** the
|
||||||
system prompt carries the ``<tools>`` section -> the deterministic
|
system prompt carries the ``<tools>`` section -> the deterministic
|
||||||
MULTI-READ flow (list → read #1 → read #2 → answer), classified by
|
MULTI-READ flow (list → drill → read #1 → read #2 → answer; phase
|
||||||
the COUNT of ``tool``-role read results (content starting with the
|
94: the drill step between the top-level listing and the first
|
||||||
agent's ``"Document <source/path>:"`` prefix); phase 70: the same
|
read), classified by the COUNT of ``tool``-role read results
|
||||||
flow on the harness-aligned names — ``ls``, then ``read`` on the
|
(content starting with the agent's ``"Document <source/path>:"``
|
||||||
JOINED combined ``source/path`` of each catalog line:
|
prefix); phase 70: the same flow on the harness-aligned names —
|
||||||
* 0 read results, no catalog yet: ``ls`` (id ``call_0``);
|
``ls``, the drill ``ls`` scoped to the first source, then ``read``
|
||||||
* 0 read results, catalog present: ``read`` on the JOINED
|
on the JOINED combined ``source/path`` of each file line:
|
||||||
combined ``source/path`` of the FIRST catalog line
|
* 0 read results, no listing yet: ``ls`` (id ``call_0``);
|
||||||
(id ``call_1``);
|
* 0 read results, top-level listing only (no file lines yet):
|
||||||
|
the drill — ``ls`` scoped to the first source (id ``call_1``);
|
||||||
|
* 0 read results, file lines present: ``read`` on the JOINED
|
||||||
|
combined ``source/path`` of the FIRST file line (id ``call_2``);
|
||||||
* 1 read result: ``read`` on the JOINED combined ``source/path``
|
* 1 read result: ``read`` on the JOINED combined ``source/path``
|
||||||
of the SECOND catalog line — the first listing line whose
|
of the SECOND file line — the first listing line whose
|
||||||
``source/path`` differs from the one already read (id
|
``source/path`` differs from the one already read (id
|
||||||
``call_2``); a one-document catalog degenerates to the
|
``call_3``); a one-file listing degenerates to the
|
||||||
single-read answer (nothing second to read);
|
single-read answer (nothing second to read);
|
||||||
* 2 read results: the forced answer, byte-stable: the single-read
|
* 2 read results: the forced answer, byte-stable: the single-read
|
||||||
shape quoting the FIRST read result, plus the line ``I read
|
shape quoting the FIRST read result, plus the line ``I read
|
||||||
@@ -162,27 +182,83 @@ Implements just enough of the aipi surface:
|
|||||||
misuse met the terse refusal and the model re-reasoned the same
|
misuse met the terse refusal and the model re-reasoned the same
|
||||||
paragraphs over and over) **and** the system prompt carries the
|
paragraphs over and over) **and** the system prompt carries the
|
||||||
``<tools>`` section -> the deterministic LS-TEACHING flow,
|
``<tools>`` section -> the deterministic LS-TEACHING flow,
|
||||||
discriminated statelessly from the messages (streaming only):
|
discriminated statelessly from the messages (streaming only;
|
||||||
|
phase 94: the correction's top-level listing carries sources only,
|
||||||
|
so the flow drills one level before the first file line exists):
|
||||||
* request 1 (``tools`` offered, no ``tool``-role result in the
|
* request 1 (``tools`` offered, no ``tool``-role result in the
|
||||||
messages yet): stream ONLY ``tool_calls`` deltas — ``ls``
|
messages yet): stream ONLY ``tool_calls`` deltas — ``ls``
|
||||||
with ``{"path": "."}`` (synthetic id ``call_0``),
|
with ``{"path": "."}`` (synthetic id ``call_0``),
|
||||||
``finish_reason: "tool_calls"``, no content — the incident's
|
``finish_reason: "tool_calls"``, no content — the incident's
|
||||||
misuse, deterministic;
|
misuse, deterministic;
|
||||||
* request 2 (a ``tool``-role result present that is NOT a
|
* request 2 (a ``tool``-role result present that is NOT a
|
||||||
catalog listing — i.e. the teaching refusal): a ``tool_calls``
|
listing with file lines — i.e. the teaching refusal): a
|
||||||
delta — ``ls`` with no arguments (id ``call_1``) — the
|
``tool_calls`` delta — ``ls`` with no arguments (id ``call_1``)
|
||||||
correction;
|
— the correction;
|
||||||
* request 3 (a ``tool``-role result whose first line matches the
|
* request 3 (the top-level source listing in the messages —
|
||||||
``^\\d+ documents:`` catalog header): a deterministic content
|
file lines still absent): the drill — a ``tool_calls`` delta
|
||||||
answer — ``These are the indexed documents: <first catalog
|
— ``ls`` scoped to the FIRST source of the listing
|
||||||
line>`` (the ``source: X | path: Y | title: Z`` line, parsed
|
(id ``call_2``);
|
||||||
with the ``_CATALOG_LINE_RE`` machinery), ``finish_reason:
|
* request 4 (a ``tool``-result with a ``source: X | path: Y |
|
||||||
"stop"`` — the loop ended in ONE correction, not at the round
|
title: Z`` file line in the messages — the folder listing):
|
||||||
|
a deterministic content answer — ``These are the indexed
|
||||||
|
documents: <first file line>`` (the line, parsed with the
|
||||||
|
``_CATALOG_LINE_RE`` machinery), ``finish_reason: "stop"`` —
|
||||||
|
the loop ended in ONE correction + ONE drill, not at the round
|
||||||
cap.
|
cap.
|
||||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (the trigger
|
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (the trigger
|
||||||
phrases are disjoint substrings — the phase-71 ordering
|
phrases are disjoint substrings — the phase-71 ordering
|
||||||
convention); no existing E2E question or fixture file contains the
|
convention); no existing E2E question or fixture file contains the
|
||||||
phrase, so every other suite is unaffected.
|
phrase, so every other suite is unaffected.
|
||||||
|
- user message containing ``drill down the tree`` (``DRILL_TRIGGER``,
|
||||||
|
phase 94 task 04 — the drill-down ``ls``'s dedicated story suite
|
||||||
|
``tests/e2e/test_ls_tree_drilldown.py``) **and** the system prompt
|
||||||
|
carries the ``<tools>`` section -> the deterministic SCRIPTED
|
||||||
|
DRILL-DOWN flow: the question carries its own tool call after the
|
||||||
|
colon — ``drill down the tree: ls [target]`` (no target = the top
|
||||||
|
level; ``target`` = a source name or ``source/folder`` path) or
|
||||||
|
``drill down the tree: read source/path`` — parsed by
|
||||||
|
``_DRILL_CALL_RE`` from the RAW user message (the target keeps its
|
||||||
|
case), then discriminated statelessly from the tool results
|
||||||
|
(streaming only):
|
||||||
|
* request 1 (``tools`` offered, no ``tool``-role result in the
|
||||||
|
messages yet): the SCRIPTED call — ``ls`` with no arguments
|
||||||
|
for the top level, ``ls``/``read`` with the parsed target
|
||||||
|
otherwise (synthetic id ``call_0``);
|
||||||
|
* the last tool result is the NOT-A-FOLDER teaching refusal
|
||||||
|
(it carries ``"is not a folder"`` — the phase-94 task-03
|
||||||
|
teaching line, the refusal being VISIBLE to the model is what
|
||||||
|
fires this branch): the scripted one-round recovery — ``ls``
|
||||||
|
the target's SOURCE segment (id ``call_1``);
|
||||||
|
* the last tool result is a READ result (``"Document
|
||||||
|
<source/path>:…``): the deterministic echo answer ``Read
|
||||||
|
<source/path>. <first 80 chars of the read document's
|
||||||
|
content>`` (the phase-37 single-read shape — the grounded-
|
||||||
|
turn citation contract);
|
||||||
|
* the last tool result is the agent's ALREADY_IN_CONTEXT dedupe
|
||||||
|
refusal (the read target is already a top-2 retrieval
|
||||||
|
document — with the drill fixture that is DETERMINISTIC:
|
||||||
|
the read question names the file's path, so the file
|
||||||
|
self-matches the hybrid gate and its FULL text is in the
|
||||||
|
``<documents>`` prompt): the model answers FROM THE PROMPT —
|
||||||
|
the deterministic answer ``Already in context: Read
|
||||||
|
<source/path>. <first 80 chars of the target document's text
|
||||||
|
as it appears in the ``<documents>`` block>`` (same citation
|
||||||
|
shape as the read-result branch — the document text reached
|
||||||
|
the model either way, and the answer proves it);
|
||||||
|
* any other last result (a top-level or folder LISTING landed):
|
||||||
|
the deterministic ECHO answer ``Here's the level I listed:\n
|
||||||
|
<the listing, verbatim>`` — the mock echoes what it received
|
||||||
|
(the house scripted-turn way of asserting on tool results):
|
||||||
|
the suite asserts on the exact tree level — the ``— N
|
||||||
|
documents`` lines, the stored folder summaries, the
|
||||||
|
``source: X | path: Y | title: Z`` file lines, and the 50-line
|
||||||
|
cap + ``…and N more documents…`` note — through the rendered
|
||||||
|
answer, the only E2E lens on the LLM's context.
|
||||||
|
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
|
||||||
|
phrases — the phase-71/72 ordering convention; the trigger needs
|
||||||
|
the ``<tools>`` section, so deflected turns never hit it); no
|
||||||
|
existing E2E question or fixture file contains the phrase, so
|
||||||
|
every other suite is unaffected.
|
||||||
- user message containing ``what are the correct llama.cpp
|
- user message containing ``what are the correct llama.cpp
|
||||||
arguments`` (``GREP_TEACH_TRIGGER``, the 2026-09-05 incident —
|
arguments`` (``GREP_TEACH_TRIGGER``, the 2026-09-05 incident —
|
||||||
the harness prior is that grep takes a REGEX; this app's grep is a
|
the harness prior is that grep takes a REGEX; this app's grep is a
|
||||||
@@ -302,7 +378,8 @@ from typing import Any
|
|||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
from fastapi.responses import JSONResponse, StreamingResponse
|
from fastapi.responses import JSONResponse, StreamingResponse
|
||||||
|
|
||||||
from app.rag.agent import CORRECTION_INSTRUCTION # phase 71: the harness constant
|
from app.rag.agent import ALREADY_IN_CONTEXT, CORRECTION_INSTRUCTION
|
||||||
|
from app.rag.folder_summaries import FOLDER_HEADER_PREFIX # phase 94: the mock's key
|
||||||
|
|
||||||
app = FastAPI()
|
app = FastAPI()
|
||||||
|
|
||||||
@@ -588,9 +665,92 @@ GREP_TEACH_PLAIN = "qwen3.8"
|
|||||||
#: plain step keys on it (a plain no-match line carries it not).
|
#: plain step keys on it (a plain no-match line carries it not).
|
||||||
GREP_TEACH_MARKER = "grep matches a plain substring"
|
GREP_TEACH_MARKER = "grep matches a plain substring"
|
||||||
|
|
||||||
#: The agent's ``ls`` listing header (app.rag.agent ``_execute_tool``):
|
# ---------------------------------------------------------------------------
|
||||||
#: ``"N documents:"`` — the first line of every catalog tool result.
|
# Phase 94 (task 04, the drill-down ls's dedicated story suite):
|
||||||
_CATALOG_HEADER_RE = re.compile(r"^\d+ documents:")
|
# the deterministic SCRIPTED drill-down turns — see the module docstring
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
#: A user message containing this substring (case-insensitive) —
|
||||||
|
#: combined with the ``<tools>`` section in the system prompt — drives
|
||||||
|
#: the scripted DRILL-DOWN flow: the question carries its own tool call
|
||||||
|
#: after the colon (``drill down the tree: ls [target]`` / ``drill
|
||||||
|
#: down the tree: read source/path`` — the suite's scripted turns,
|
||||||
|
#: ``tests/e2e/test_ls_tree_drilldown.py``). Checked BEFORE the plain
|
||||||
|
#: ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the phase-71/72
|
||||||
|
#: ordering convention); verified: no existing E2E question or fixture
|
||||||
|
#: file contains the phrase, so every other suite is unaffected.
|
||||||
|
DRILL_TRIGGER = "drill down the tree"
|
||||||
|
|
||||||
|
#: The scripted call in the drill-down question (case-insensitive — the
|
||||||
|
#: suite's questions capitalize the trigger's first letter): the verb
|
||||||
|
#: (``ls`` / ``read``) plus the optional target — a source name or
|
||||||
|
#: ``source/folder`` path (``ls``) or a combined ``source/path``
|
||||||
|
#: (``read``), parsed from the RAW user message so the target keeps its
|
||||||
|
#: case. The target is a ``[a-z0-9_./-]`` run (case-insensitively), so
|
||||||
|
#: the suite's `` — `` flavor separator (em dash) can never bleed into
|
||||||
|
#: it; no run = the top-level ``ls`` (no ``path`` argument).
|
||||||
|
_DRILL_CALL_RE = re.compile(
|
||||||
|
r"drill down the tree:\s*(?P<verb>ls|read)(?:\s+(?P<arg>[a-z0-9_./-]+))?",
|
||||||
|
re.I,
|
||||||
|
)
|
||||||
|
|
||||||
|
#: The agent's NOT-A-FOLDER teaching refusal marker (app.rag.agent
|
||||||
|
#: ``NOT_A_FOLDER`` — ``'{arg}' is not a folder — {parent} has:
|
||||||
|
#: {subfolders}``): the drill-down flow's scripted recovery keys on it
|
||||||
|
#: in the LAST tool result (the refusal being visible to the model is
|
||||||
|
#: exactly what triggers the one-round recovery — the phase-72
|
||||||
|
#: self-correction contract carrying the tree's teaching line).
|
||||||
|
_NOT_A_FOLDER_MARKER = "is not a folder"
|
||||||
|
|
||||||
|
#: The stable substring of the harness-owned dedupe refusal the
|
||||||
|
#: drill-down flow's ``ctx_answer`` branch keys on (a read target that
|
||||||
|
#: is already a top-2 retrieval document — the mock then answers from
|
||||||
|
#: the document's text in the ``<documents>`` prompt block, exactly
|
||||||
|
#: what the refusal instructs). Keyed on a substring (not the whole
|
||||||
|
#: constant) so a re-wrap of the constant cannot silently re-route the
|
||||||
|
#: mock; the module-level assert below fails loudly if the substring
|
||||||
|
#: ever leaves the constant (the mock must never drift from
|
||||||
|
#: ``app.rag.agent.ALREADY_IN_CONTEXT``).
|
||||||
|
_ALREADY_IN_CONTEXT_MARKER = "Already in your context"
|
||||||
|
assert _ALREADY_IN_CONTEXT_MARKER in ALREADY_IN_CONTEXT, (
|
||||||
|
"mock drift: the dedupe marker left ALREADY_IN_CONTEXT"
|
||||||
|
)
|
||||||
|
|
||||||
|
#: One ``<document>`` block of the HIGH prompt's ``<documents>``
|
||||||
|
#: section (``app.rag.prompts.build_high_prompt``): the block is the
|
||||||
|
#: document identity (``source``/``path``/``title`` attributes) plus
|
||||||
|
#: the document's FULL text (never truncated on the retrieval path,
|
||||||
|
#: owner-locked A7) between the tags.
|
||||||
|
_DOCUMENT_BLOCK_RE = re.compile(
|
||||||
|
r'<document source="(?P<source>[^"]+)" path="(?P<path>[^"]+)" '
|
||||||
|
r'title="[^"]*">\n(?P<content>.*?)\n</document>',
|
||||||
|
re.S,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _document_block(system: str, source: str, path: str) -> str | None:
|
||||||
|
"""The stored text of one ``<document>`` block (or ``None``).
|
||||||
|
|
||||||
|
The drill-down flow's ``ctx_answer`` branch: when the agent's read
|
||||||
|
of a top-2 retrieval document gets the ALREADY_IN_CONTEXT dedupe,
|
||||||
|
the document's full text is in the ``<documents>`` prompt — the
|
||||||
|
mock (the model) extracts it by the block's identity attributes
|
||||||
|
and quotes it, answering from the prompt as the refusal instructs.
|
||||||
|
"""
|
||||||
|
for block in _DOCUMENT_BLOCK_RE.finditer(system):
|
||||||
|
if block.group("source") == source and block.group("path") == path:
|
||||||
|
return block.group("content")
|
||||||
|
return None
|
||||||
|
|
||||||
|
#: The agent's drill-down ``ls`` result shapes (app.rag.agent
|
||||||
|
#: ``render_ls_top`` / ``render_folder_listing``, phase 94): the
|
||||||
|
#: top-level header ``"N sources:"`` and the per-source block line
|
||||||
|
#: ``"<source> — N documents"`` (summary lines are indented — they
|
||||||
|
#: never match); the folder-level header ``"<identity> — N documents,
|
||||||
|
#: M folders:"`` (identity = the source name or ``source/folder``).
|
||||||
|
_SOURCE_HEADER_RE = re.compile(r"^\d+ sources:$")
|
||||||
|
_SOURCE_LINE_RE = re.compile(r"^(?P<source>.+?) — (?P<n>\d+) documents$")
|
||||||
|
_FOLDER_HEADER_RE = re.compile(r"^(?P<identity>.+?) — \d+ documents, \d+ folders:$")
|
||||||
|
|
||||||
#: One DEAD app-level chat attempt costs exactly this many HTTP POSTs
|
#: One DEAD app-level chat attempt costs exactly this many HTTP POSTs
|
||||||
#: while the endpoint stays down: the openai SDK's default policy
|
#: while the endpoint stays down: the openai SDK's default policy
|
||||||
@@ -713,26 +873,60 @@ def _tool_results(body: dict[str, Any]) -> list[str]:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
def _first_catalog_line(body: dict[str, Any]) -> str | None:
|
def _first_file_line(body: dict[str, Any]) -> str | None:
|
||||||
"""The first catalog line of a catalog listing in the messages.
|
"""The first file line (``source: X | path: Y | title: Z``) across
|
||||||
|
the ``tool``-role results in the messages, in message order (read
|
||||||
|
results — full documents, not listings — skipped; the ``_CATALOG_-
|
||||||
|
LINE_RE`` machinery).
|
||||||
|
|
||||||
A catalog listing is a ``tool``-role result whose FIRST line is the
|
Phase 94: the drill-down ``ls`` carries file lines only at folder
|
||||||
agent's ``"N documents:"`` header (``_CATALOG_HEADER_RE``); its
|
levels — the top-level source listing and the teaching refusals
|
||||||
first ``source: X | path: Y | title: Z`` line (the
|
have none, so ``None`` here means "no folder level reached yet".
|
||||||
``_CATALOG_LINE_RE`` machinery) is returned. ``None`` when no
|
|
||||||
catalog listing is in the messages — e.g. while only the teaching
|
|
||||||
refusal is there (the phase-72 LS-TEACH flow's request-2 state).
|
|
||||||
An empty listing (``"0 documents:"`` with no lines) returns
|
|
||||||
``""`` — the listing is present, it is just empty.
|
|
||||||
"""
|
"""
|
||||||
for content in _tool_results(body):
|
for content in _tool_results(body):
|
||||||
lines = content.splitlines()
|
if content.startswith(_READ_RESULT_PREFIX):
|
||||||
if not lines or not _CATALOG_HEADER_RE.match(lines[0]):
|
|
||||||
continue
|
continue
|
||||||
for line in lines[1:]:
|
for line in content.splitlines():
|
||||||
if _CATALOG_LINE_RE.match(line):
|
if _CATALOG_LINE_RE.match(line):
|
||||||
return line
|
return line
|
||||||
return ""
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _drill_target(body: dict[str, Any]) -> str | None:
|
||||||
|
"""The next source to drill into (phase 94, the drill step).
|
||||||
|
|
||||||
|
The drill-down ``ls`` top level lists SOURCES only (no file lines),
|
||||||
|
so a deterministic flow that needs a file line must drill one level:
|
||||||
|
``ls`` scoped to a source. The target is the FIRST source of the
|
||||||
|
top-level listing (registry order — the listing's own order) that
|
||||||
|
does not yet have a folder-level listing in the messages (a folder
|
||||||
|
header whose identity is the source or ``source/…``); ``None`` when
|
||||||
|
no top-level listing is in the messages yet (nothing to drill from)
|
||||||
|
or every listed source has been drilled (an empty KB — the flow
|
||||||
|
degenerates to the old re-list loop, settling at the round cap
|
||||||
|
exactly like the phase-70 empty-catalog case).
|
||||||
|
"""
|
||||||
|
sources: list[str] = []
|
||||||
|
drilled: set[str] = set()
|
||||||
|
for content in _tool_results(body):
|
||||||
|
lines = content.splitlines()
|
||||||
|
if not lines:
|
||||||
|
continue
|
||||||
|
if _SOURCE_HEADER_RE.match(lines[0]):
|
||||||
|
for line in lines[1:]:
|
||||||
|
match = _SOURCE_LINE_RE.match(line)
|
||||||
|
if match:
|
||||||
|
sources.append(match.group("source"))
|
||||||
|
else:
|
||||||
|
folder = _FOLDER_HEADER_RE.match(lines[0])
|
||||||
|
if folder:
|
||||||
|
drilled.add(folder.group("identity"))
|
||||||
|
for source in sources:
|
||||||
|
if not any(
|
||||||
|
identity == source or identity.startswith(source + "/")
|
||||||
|
for identity in drilled
|
||||||
|
):
|
||||||
|
return source
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
@@ -853,11 +1047,14 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
|||||||
``TOOLS_TRIGGER`` and ``MULTI_READ_TRIGGER``), classified by the
|
``TOOLS_TRIGGER`` and ``MULTI_READ_TRIGGER``), classified by the
|
||||||
count of ``tool``-role read results:
|
count of ``tool``-role read results:
|
||||||
|
|
||||||
* 0 read results: ``("list", "", "")`` (no catalog yet) or
|
* 0 read results: ``("list", "", "")`` (no listing yet),
|
||||||
``("read", source, path, "call_1")`` on the FIRST catalog doc.
|
``("drill", source, "call_1")`` when the top-level source
|
||||||
* 1 read result: ``("read", source, path, "call_2")`` on the SECOND
|
listing is in the messages but no file line yet (phase 94: the
|
||||||
catalog doc — the first listing line whose ``source/path``
|
drill — the top level carries sources only), or
|
||||||
differs from the one already read. A one-document catalog
|
``("read", source, path, "call_2")`` on the FIRST file-line doc.
|
||||||
|
* 1 read result: ``("read", source, path, "call_3")`` on the SECOND
|
||||||
|
file-line doc — the first listing line whose ``source/path``
|
||||||
|
differs from the one already read. A one-file listing
|
||||||
degenerates to the single-read ``("answer", ...)`` shape (nothing
|
degenerates to the single-read ``("answer", ...)`` shape (nothing
|
||||||
second to read).
|
second to read).
|
||||||
* 2 read results: ``("multi_answer", "", text)`` — the forced
|
* 2 read results: ``("multi_answer", "", text)`` — the forced
|
||||||
@@ -880,30 +1077,37 @@ def _tool_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
|||||||
if not body.get("tools"):
|
if not body.get("tools"):
|
||||||
return None
|
return None
|
||||||
docs = _catalog_docs(body)
|
docs = _catalog_docs(body)
|
||||||
if not docs:
|
if docs:
|
||||||
return ("list", "", "")
|
return ("read", docs[0][0], docs[0][1], "call_2")
|
||||||
return ("read", docs[0][0], docs[0][1], "call_1")
|
drill = _drill_target(body)
|
||||||
|
if drill is not None:
|
||||||
|
return ("drill", drill, "call_1")
|
||||||
|
return ("list", "", "")
|
||||||
if len(reads) == 1:
|
if len(reads) == 1:
|
||||||
skip = reads[0][0]
|
skip = reads[0][0]
|
||||||
second = next(
|
second = next(
|
||||||
(d for d in _catalog_docs(body) if f"{d[0]}/{d[1]}" != skip), None
|
(d for d in _catalog_docs(body) if f"{d[0]}/{d[1]}" != skip), None
|
||||||
)
|
)
|
||||||
if second is None:
|
if second is None:
|
||||||
# One-document catalog: nothing second to read — the
|
# One-file listing: nothing second to read — the
|
||||||
# single-read answer shape (deterministic degenerate).
|
# single-read answer shape (deterministic degenerate).
|
||||||
return ("answer", reads[0][0], reads[0][1])
|
return ("answer", reads[0][0], reads[0][1])
|
||||||
return ("read", second[0], second[1], "call_2")
|
return ("read", second[0], second[1], "call_3")
|
||||||
(sp1, c1), (sp2, _c2) = reads[0], reads[1]
|
(sp1, c1), (sp2, _c2) = reads[0], reads[1]
|
||||||
answer = f"Read {sp1}. {c1[:80]} I read {sp1} and {sp2}."
|
answer = f"Read {sp1}. {c1[:80]} I read {sp1} and {sp2}."
|
||||||
return ("multi_answer", "", answer)
|
return ("multi_answer", "", answer)
|
||||||
# Phase-37 single-read flow — byte-identical to the original.
|
# Phase-37 single-read flow (phase 94: the drill step between the
|
||||||
|
# top-level listing and the first file line).
|
||||||
if reads:
|
if reads:
|
||||||
return ("answer", reads[0][0], reads[0][1])
|
return ("answer", reads[0][0], reads[0][1])
|
||||||
if not body.get("tools"):
|
if not body.get("tools"):
|
||||||
return None
|
return None
|
||||||
docs = _catalog_docs(body)
|
docs = _catalog_docs(body)
|
||||||
if docs:
|
if docs:
|
||||||
return ("read", docs[0][0], docs[0][1], "call_1")
|
return ("read", docs[0][0], docs[0][1], "call_2")
|
||||||
|
drill = _drill_target(body)
|
||||||
|
if drill is not None:
|
||||||
|
return ("drill", drill, "call_1")
|
||||||
return ("list", "", "")
|
return ("list", "", "")
|
||||||
|
|
||||||
|
|
||||||
@@ -915,13 +1119,17 @@ def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
|||||||
with ``{"path": "."}`` (id ``call_0``), ``finish_reason:
|
with ``{"path": "."}`` (id ``call_0``), ``finish_reason:
|
||||||
"tool_calls"``, no content.
|
"tool_calls"``, no content.
|
||||||
* ``("correct",)`` — a ``tool``-role result is in the messages and
|
* ``("correct",)`` — a ``tool``-role result is in the messages and
|
||||||
it is NOT a catalog listing (the teaching refusal): the
|
no file line exists yet (the teaching refusal): the correction —
|
||||||
correction — ``ls`` with no arguments (id ``call_1``).
|
``ls`` with no arguments (id ``call_1``).
|
||||||
* ``("answer", line)`` — a ``tool``-role result whose first line
|
* ``("drill", source, "call_2")`` — the top-level source listing
|
||||||
is the ``"N documents:"`` catalog header: the deterministic
|
is in the messages (the correction ran) but no file line yet
|
||||||
|
(phase 94: the top level carries sources only): the drill —
|
||||||
|
``ls`` scoped to the first source of the listing.
|
||||||
|
* ``("answer", line)`` — a ``source: X | path: Y | title: Z`` file
|
||||||
|
line is in the messages (the folder listing): the deterministic
|
||||||
content answer ``These are the indexed documents: <line>`` (the
|
content answer ``These are the indexed documents: <line>`` (the
|
||||||
first catalog line), ``finish_reason: "stop"`` — the loop
|
first file line), ``finish_reason: "stop"`` — the loop settled
|
||||||
settled in ONE correction, not at the round cap.
|
in ONE correction + ONE drill, not at the round cap.
|
||||||
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
|
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
|
||||||
section is missing (deflected turns never carry it), or
|
section is missing (deflected turns never carry it), or
|
||||||
``tools`` are not offered and no tool results are in the
|
``tools`` are not offered and no tool results are in the
|
||||||
@@ -931,9 +1139,12 @@ def _ls_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
|||||||
return None
|
return None
|
||||||
if "<tools>" not in _system(body):
|
if "<tools>" not in _system(body):
|
||||||
return None
|
return None
|
||||||
line = _first_catalog_line(body)
|
line = _first_file_line(body)
|
||||||
if line is not None:
|
if line is not None:
|
||||||
return ("answer", line)
|
return ("answer", line)
|
||||||
|
drill = _drill_target(body)
|
||||||
|
if drill is not None:
|
||||||
|
return ("drill", drill, "call_2")
|
||||||
if _tool_results(body):
|
if _tool_results(body):
|
||||||
return ("correct",)
|
return ("correct",)
|
||||||
if not body.get("tools"):
|
if not body.get("tools"):
|
||||||
@@ -996,6 +1207,92 @@ def _grep_teach_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
|||||||
return ("nomatch",)
|
return ("nomatch",)
|
||||||
|
|
||||||
|
|
||||||
|
def _drill_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||||
|
"""Classify a phase-94 scripted drill-down request (see the module
|
||||||
|
docstring). The question carries the scripted call
|
||||||
|
(``drill down the tree: ls [target]`` / ``drill down the tree: read
|
||||||
|
source/path``); the step is then discriminated statelessly from the
|
||||||
|
tool results, like the other marker flows:
|
||||||
|
|
||||||
|
* ``("call", verb, target, "call_0")`` — ``tools`` are offered and
|
||||||
|
no ``tool``-role result is in the messages yet: the scripted call
|
||||||
|
(``ls`` with NO path for the top level — empty target —, ``ls``
|
||||||
|
with the target for source/folder levels, ``read`` with the
|
||||||
|
combined ``source/path``).
|
||||||
|
* ``("recover", source)`` — the LAST tool result is the NOT-A-FOLDER
|
||||||
|
teaching refusal (it carries :data:`_NOT_A_FOLDER_MARKER`): the
|
||||||
|
scripted one-round recovery — ``ls`` the target's SOURCE segment
|
||||||
|
(the refusal listing the parent's subfolders is what lets the
|
||||||
|
model — and this mock — self-correct in the next round, the
|
||||||
|
phase-72 contract).
|
||||||
|
* ``("read_answer", combined, quote)`` — the LAST tool result is a
|
||||||
|
read result (``"Document <source/path>:\n<content>"``): the
|
||||||
|
deterministic echo answer ``Read <source/path>. <first 80 chars>
|
||||||
|
`` (the phase-37 single-read shape — the grounded-turn citation
|
||||||
|
contract the suite asserts).
|
||||||
|
* ``("ctx_answer", target, quote)`` — the scripted call was a
|
||||||
|
``read`` and the LAST tool result is the ALREADY_IN_CONTEXT
|
||||||
|
dedupe refusal (the target is already a top-2 retrieval document
|
||||||
|
— deterministic for the drill fixture: the read question names
|
||||||
|
the file's path, so the file self-matches the hybrid gate): the
|
||||||
|
model answers FROM THE PROMPT — ``Already in context: Read
|
||||||
|
<target>. <first 80 chars of the target document's text in the
|
||||||
|
``<documents>`` block>`` (the same citation shape as the
|
||||||
|
read-result branch — the document text reached the model either
|
||||||
|
way). Falls through to the echo when the target is not a
|
||||||
|
``<document>`` block (the premise broke — the suite fails
|
||||||
|
loudly on the answer).
|
||||||
|
* ``("echo", listing)`` — a listing landed (top-level or folder —
|
||||||
|
any other last result): the deterministic ECHO — the answer
|
||||||
|
carries the listing VERBATIM (``Here's the level I listed:\n
|
||||||
|
<listing>``), so the suite asserts on the exact tree level the
|
||||||
|
model saw (source ``— N documents`` lines + stored summaries,
|
||||||
|
subfolder lines, the ``source: X | path: Y | title: Z`` file
|
||||||
|
lines, the 50-line cap + note) through the rendered answer — the
|
||||||
|
house scripted-turn way of asserting on tool results (the mock is
|
||||||
|
the only E2E lens on the LLM's context).
|
||||||
|
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
|
||||||
|
section is missing (deflected turns never carry it), the scripted
|
||||||
|
call is unparseable, or ``tools`` are not offered and no tool
|
||||||
|
results are in the messages yet (e.g. ``agent_max_rounds=0``).
|
||||||
|
"""
|
||||||
|
user = _user(body)
|
||||||
|
if DRILL_TRIGGER not in user.lower():
|
||||||
|
return None
|
||||||
|
if "<tools>" not in _system(body):
|
||||||
|
return None
|
||||||
|
match = _DRILL_CALL_RE.search(user)
|
||||||
|
if match is None:
|
||||||
|
return None
|
||||||
|
verb = match.group("verb")
|
||||||
|
target = match.group("arg") or ""
|
||||||
|
results = _tool_results(body)
|
||||||
|
if not results:
|
||||||
|
if not body.get("tools"):
|
||||||
|
return None
|
||||||
|
return ("call", verb, target, "call_0")
|
||||||
|
last = results[-1]
|
||||||
|
if last.startswith(_READ_RESULT_PREFIX):
|
||||||
|
head, _, content = last.partition("\n")
|
||||||
|
# The read result is ``"Document <source/path>:\n<content>"`` —
|
||||||
|
# the head carries the server's appended ``:`` (removed here;
|
||||||
|
# a document path never legitimately ends with one).
|
||||||
|
combined = head[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
|
||||||
|
return ("read_answer", combined, content[:80])
|
||||||
|
if _ALREADY_IN_CONTEXT_MARKER in last and verb == "read" and "/" in target:
|
||||||
|
# The dedupe fired: the read target is already a top-2
|
||||||
|
# retrieval document, so its FULL text is in the
|
||||||
|
# ``<documents>`` prompt — answer from the prompt (the
|
||||||
|
# refusal's instruction), quoting the block's text.
|
||||||
|
src, _, p = target.partition("/")
|
||||||
|
content = _document_block(_system(body), src, p)
|
||||||
|
if content is not None:
|
||||||
|
return ("ctx_answer", target, content[:80])
|
||||||
|
if _NOT_A_FOLDER_MARKER in last:
|
||||||
|
return ("recover", target.split("/")[0])
|
||||||
|
return ("echo", last)
|
||||||
|
|
||||||
|
|
||||||
def long_answer() -> str:
|
def long_answer() -> str:
|
||||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||||
a unique final line that must survive the stream untruncated."""
|
a unique final line that must survive the stream untruncated."""
|
||||||
@@ -1103,6 +1400,28 @@ def compose_answer(body: dict[str, Any]) -> str:
|
|||||||
user = _user(body)
|
user = _user(body)
|
||||||
if LONG_ANSWER_TRIGGER in user.lower():
|
if LONG_ANSWER_TRIGGER in user.lower():
|
||||||
answer = long_answer()
|
answer = long_answer()
|
||||||
|
elif "FOLDER_SUMMARY_MODE" in system:
|
||||||
|
# Folder summary (phase 94, TODO.md L4): the ``lite`` stand-in
|
||||||
|
# returns the deterministic one-liner
|
||||||
|
# ``Fixture folder summary for <folder>.`` — <folder> is the
|
||||||
|
# user message's ``Folder: …`` header line (the generator puts
|
||||||
|
# the canonical source/folder identity there, imported as
|
||||||
|
# ``FOLDER_HEADER_PREFIX`` so the mock can never drift from it).
|
||||||
|
# The stored row therefore always names its folder — the drill-
|
||||||
|
# down E2E (``test_ls_tree_drilldown.py``) asserts on it.
|
||||||
|
# Checked BEFORE the ``SUMMARY_MODE`` branch: the folder marker
|
||||||
|
# CONTAINS the summary marker as a substring (``FOLDER_`` +
|
||||||
|
# ``SUMMARY_MODE``), so the summary branch would otherwise
|
||||||
|
# shadow every folder-summary call. Checked BEFORE the
|
||||||
|
# DEFLECT_MODE branch, like the other lite-mode markers (a
|
||||||
|
# deflection prompt never carries one).
|
||||||
|
header = user.splitlines()[0] if user else ""
|
||||||
|
folder = (
|
||||||
|
header.removeprefix(FOLDER_HEADER_PREFIX).strip()
|
||||||
|
if header.startswith(FOLDER_HEADER_PREFIX)
|
||||||
|
else "(unnamed folder)"
|
||||||
|
)
|
||||||
|
answer = f"Fixture folder summary for {folder}."
|
||||||
elif "SUMMARY_MODE" in system:
|
elif "SUMMARY_MODE" in system:
|
||||||
# Document summaries (phase 30): the ``lite`` stand-in returns a
|
# Document summaries (phase 30): the ``lite`` stand-in returns a
|
||||||
# deterministic digest — the first 24 tokens of the user message
|
# deterministic digest — the first 24 tokens of the user message
|
||||||
@@ -1594,9 +1913,16 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
|||||||
# The incident's misuse, deterministic: ls(path='.').
|
# The incident's misuse, deterministic: ls(path='.').
|
||||||
stream = _tool_call_stream("ls", {"path": "."}, "call_0")
|
stream = _tool_call_stream("ls", {"path": "."}, "call_0")
|
||||||
elif ls_teach[0] == "correct":
|
elif ls_teach[0] == "correct":
|
||||||
# The one-round correction: the no-arg full listing.
|
# The one-round correction: the no-arg top-level
|
||||||
|
# listing (phase 94: sources only — the drill follows).
|
||||||
stream = _tool_call_stream("ls", {}, "call_1")
|
stream = _tool_call_stream("ls", {}, "call_1")
|
||||||
else: # "answer" — quote the first catalog line
|
elif ls_teach[0] == "drill":
|
||||||
|
# Phase 94: the top level lists sources only — drill
|
||||||
|
# one level into the first source for the file lines.
|
||||||
|
stream = _tool_call_stream(
|
||||||
|
"ls", {"path": ls_teach[1]}, ls_teach[2]
|
||||||
|
)
|
||||||
|
else: # "answer" — quote the first file line
|
||||||
answer = _apply_max_tokens(
|
answer = _apply_max_tokens(
|
||||||
f"These are the indexed documents: {ls_teach[1]}",
|
f"These are the indexed documents: {ls_teach[1]}",
|
||||||
body.get("max_tokens"),
|
body.get("max_tokens"),
|
||||||
@@ -1607,10 +1933,67 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
|||||||
media_type="text/event-stream",
|
media_type="text/event-stream",
|
||||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||||
)
|
)
|
||||||
|
# Phase 94 (task 04): the deterministic SCRIPTED drill-down
|
||||||
|
# turns (sources → folders → files → the grounded read; the
|
||||||
|
# 50-line cap; the NOT-A-FOLDER teaching + scripted recovery) —
|
||||||
|
# checked BEFORE the plain TOOLS_TRIGGER flow (disjoint trigger
|
||||||
|
# phrases — the phase-71/72 ordering convention; the trigger
|
||||||
|
# needs the ``<tools>`` section, so deflected turns never hit
|
||||||
|
# it).
|
||||||
|
drill = _drill_flow(body)
|
||||||
|
if drill is not None:
|
||||||
|
if drill[0] == "call":
|
||||||
|
# The scripted call: ls with no path at the top level
|
||||||
|
# (empty target), ls/read with the parsed target.
|
||||||
|
stream = _tool_call_stream(
|
||||||
|
drill[1], {"path": drill[2]} if drill[2] else {}, drill[3]
|
||||||
|
)
|
||||||
|
elif drill[0] == "recover":
|
||||||
|
# The scripted one-round recovery after the NOT-A-FOLDER
|
||||||
|
# teaching: ls the target's source (the parent level).
|
||||||
|
stream = _tool_call_stream("ls", {"path": drill[1]}, "call_1")
|
||||||
|
elif drill[0] == "read_answer":
|
||||||
|
# Quote the read document (first 80 chars) — the
|
||||||
|
# phase-37 single-read shape.
|
||||||
|
stream = _sse_stream(
|
||||||
|
_apply_max_tokens(
|
||||||
|
f"Read {drill[1]}. {drill[2]}", body.get("max_tokens")
|
||||||
|
),
|
||||||
|
0.0,
|
||||||
|
)
|
||||||
|
elif drill[0] == "ctx_answer":
|
||||||
|
# The dedupe fired: the target's full text is in the
|
||||||
|
# <documents> prompt — answer from it (same citation
|
||||||
|
# shape, prefixed so the suite pins the dedupe path).
|
||||||
|
stream = _sse_stream(
|
||||||
|
_apply_max_tokens(
|
||||||
|
f"Already in context: Read {drill[1]}. {drill[2]}",
|
||||||
|
body.get("max_tokens"),
|
||||||
|
),
|
||||||
|
0.0,
|
||||||
|
)
|
||||||
|
else: # "echo" — the listing verbatim (the suite's lens)
|
||||||
|
stream = _sse_stream(
|
||||||
|
_apply_max_tokens(
|
||||||
|
f"Here's the level I listed:\n{drill[1]}",
|
||||||
|
body.get("max_tokens"),
|
||||||
|
),
|
||||||
|
0.0,
|
||||||
|
)
|
||||||
|
return StreamingResponse(
|
||||||
|
stream,
|
||||||
|
media_type="text/event-stream",
|
||||||
|
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||||
|
)
|
||||||
flow = _tool_flow(body)
|
flow = _tool_flow(body)
|
||||||
if flow is not None:
|
if flow is not None:
|
||||||
if flow[0] == "list":
|
if flow[0] == "list":
|
||||||
stream = _tool_call_stream("ls", {}, "call_0")
|
stream = _tool_call_stream("ls", {}, "call_0")
|
||||||
|
elif flow[0] == "drill":
|
||||||
|
# Phase 94: the drill step — ls scoped to the first
|
||||||
|
# source of the top-level listing (flow[1], flow[2] is
|
||||||
|
# the synthetic call id).
|
||||||
|
stream = _tool_call_stream("ls", {"path": flow[1]}, flow[2])
|
||||||
elif flow[0] == "read":
|
elif flow[0] == "read":
|
||||||
# flow[3] is the synthetic call id — "call_1" for the
|
# flow[3] is the synthetic call id — "call_1" for the
|
||||||
# single-read flow and the multi-read first read,
|
# single-read flow and the multi-read first read,
|
||||||
|
|||||||
@@ -11,15 +11,22 @@ deterministic marker flow in ``tests/e2e/mock_llm.py`` (user message
|
|||||||
contains ``use your tools`` **and** the system prompt carries the
|
contains ``use your tools`` **and** the system prompt carries the
|
||||||
``<tools>`` section of the HIGH prompt; phase 70: the flow emits the
|
``<tools>`` section of the HIGH prompt; phase 70: the flow emits the
|
||||||
harness-aligned names — ``ls`` / ``read`` with the combined
|
harness-aligned names — ``ls`` / ``read`` with the combined
|
||||||
``source/path`` identity):
|
``source/path`` identity; phase 94: the drill-down ``ls`` — the top
|
||||||
|
level lists sources only, so the flow drills one level into the first
|
||||||
|
source before the first file line exists):
|
||||||
|
|
||||||
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
||||||
``tool_calls`` deltas calling ``ls`` (id ``call_0``, no arguments,
|
``tool_calls`` deltas calling ``ls`` (id ``call_0``, no arguments,
|
||||||
``finish_reason: "tool_calls"``);
|
``finish_reason: "tool_calls"``);
|
||||||
2. request 2 (a ``tool``-role catalog result in the messages) → streams a
|
2. request 2 (the top-level source listing in the messages — no file
|
||||||
``tool_calls`` delta calling ``read`` on the JOINED combined
|
lines yet) → a ``tool_calls`` delta — ``ls`` scoped to the FIRST
|
||||||
``source/path`` of the FIRST catalog line (id ``call_1``);
|
source of the listing (id ``call_1``) — the drill step (the seed
|
||||||
3. request 3 (a ``tool``-role read result in the messages) → the content
|
registers ``Deployments`` first, so the drill — and therefore the
|
||||||
|
read — lands on the JSON file);
|
||||||
|
3. request 3 (a ``tool``-role folder listing with file lines) → streams
|
||||||
|
a ``tool_calls`` delta calling ``read`` on the JOINED combined
|
||||||
|
``source/path`` of the FIRST file line (id ``call_2``);
|
||||||
|
4. request 4 (a ``tool``-role read result in the messages) → the content
|
||||||
answer ``Read <source/path>. <first 80 chars of the read document's
|
answer ``Read <source/path>. <first 80 chars of the read document's
|
||||||
content>`` — so the suite can assert the read document reached the
|
content>`` — so the suite can assert the read document reached the
|
||||||
model and landed in the answer.
|
model and landed in the answer.
|
||||||
@@ -72,7 +79,7 @@ from sqlalchemy import select, text
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from app.db import SessionLocal
|
from app.db import SessionLocal
|
||||||
from app.models import Chunk, Document, QueryLog
|
from app.models import Chunk, Document, GitSource, QueryLog
|
||||||
from e2e.auth_helpers import login
|
from e2e.auth_helpers import login
|
||||||
from tests.e2e.mock_llm import embed_text
|
from tests.e2e.mock_llm import embed_text
|
||||||
|
|
||||||
@@ -148,7 +155,22 @@ READ_CHIP_HREF = f"/document.html?source={READ_SOURCE}&path={READ_PATH}&back=%2F
|
|||||||
|
|
||||||
|
|
||||||
def _seed(db: Session) -> None:
|
def _seed(db: Session) -> None:
|
||||||
"""The two-document pair from the TODO (see the module docstring)."""
|
"""The two-document pair from the TODO (see the module docstring).
|
||||||
|
|
||||||
|
Phase 94: the drill-down ``ls`` top level reads the registry — the
|
||||||
|
seed registers BOTH sources (TRUNCATEd in ``_reset_db``),
|
||||||
|
``Deployments`` FIRST: registry order is ``(added_at, id)``, so the
|
||||||
|
mock's drill (first source of the listing) lands on the JSON file
|
||||||
|
deterministically — independent of the operator's
|
||||||
|
``BOR_GIT_SOURCES`` (a non-empty table ignores the env fallback).
|
||||||
|
"""
|
||||||
|
# COMMIT between the inserts (not flush): ``added_at`` is
|
||||||
|
# ``server_default now()`` — the transaction timestamp — and the
|
||||||
|
# tie-break is the random uuid ``id``, so one-transaction rows order
|
||||||
|
# nondeterministically.
|
||||||
|
db.add(GitSource(url=READ_SOURCE, kind="local"))
|
||||||
|
db.commit()
|
||||||
|
db.add(GitSource(url=SEED_SOURCE, kind="local"))
|
||||||
md = Document(
|
md = Document(
|
||||||
source=SEED_SOURCE,
|
source=SEED_SOURCE,
|
||||||
path=SEED_PATH,
|
path=SEED_PATH,
|
||||||
@@ -195,7 +217,10 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
|
|||||||
"""
|
"""
|
||||||
with SessionLocal() as db:
|
with SessionLocal() as db:
|
||||||
db.execute(
|
db.execute(
|
||||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
text(
|
||||||
|
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||||
|
"kb_overview, git_sources"
|
||||||
|
)
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
if seed is not None:
|
if seed is not None:
|
||||||
@@ -389,12 +414,14 @@ def test_marker_question_lists_reads_and_quotes(
|
|||||||
assert i_list is not None and i_read is not None, statuses
|
assert i_list is not None and i_read is not None, statuses
|
||||||
assert i_list < i_read, statuses
|
assert i_list < i_read, statuses
|
||||||
|
|
||||||
# Wire level: exactly two `tool` frames — ``ls`` then ``read`` (the
|
# Wire level: exactly three `tool` frames — ``ls`` (the top level),
|
||||||
# combined source/path as the model passed it) — and both ahead of
|
# the drill ``ls`` scoped to the first source (phase 94), then
|
||||||
# the first `delta` frame.
|
# ``read`` (the combined source/path as the model passed it) — and
|
||||||
|
# all three ahead of the first `delta` frame.
|
||||||
frames = _frames(page)
|
frames = _frames(page)
|
||||||
assert _tool_frames(frames) == [
|
assert _tool_frames(frames) == [
|
||||||
{"type": "tool", "name": "ls", "argument": None},
|
{"type": "tool", "name": "ls", "argument": None},
|
||||||
|
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
|
||||||
{"type": "tool", "name": "read", "argument": READ_SP},
|
{"type": "tool", "name": "read", "argument": READ_SP},
|
||||||
]
|
]
|
||||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||||
@@ -408,12 +435,15 @@ def test_marker_question_lists_reads_and_quotes(
|
|||||||
(READ_SOURCE, READ_PATH),
|
(READ_SOURCE, READ_PATH),
|
||||||
]
|
]
|
||||||
|
|
||||||
# Both tool lines, in order, above the answer.
|
# All three tool lines, in order, above the answer (phase 94: the
|
||||||
|
# drill line is "Listing documents in <source>").
|
||||||
lines = page.locator(".msg.brain .tool-call")
|
lines = page.locator(".msg.brain .tool-call")
|
||||||
expect(lines).to_have_count(2)
|
expect(lines).to_have_count(3)
|
||||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||||
expect(lines.nth(1)).to_contain_text(READ_SP)
|
expect(lines.nth(1)).to_contain_text(READ_SOURCE)
|
||||||
|
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||||
|
expect(lines.nth(2)).to_contain_text(READ_SP)
|
||||||
|
|
||||||
# The final answer quotes the read document (the mock's deterministic
|
# The final answer quotes the read document (the mock's deterministic
|
||||||
# quote: "Read <source/path>. <first 80 chars of its content>").
|
# quote: "Read <source/path>. <first 80 chars of its content>").
|
||||||
@@ -451,18 +481,20 @@ def test_tool_lines_re_render_after_reload(
|
|||||||
|
|
||||||
_submit(page, MARKER_QUESTION)
|
_submit(page, MARKER_QUESTION)
|
||||||
_wait_settled(page)
|
_wait_settled(page)
|
||||||
expect(page.locator(".msg.brain .tool-call")).to_have_count(2)
|
expect(page.locator(".msg.brain .tool-call")).to_have_count(3)
|
||||||
|
|
||||||
page.reload()
|
page.reload()
|
||||||
expect(page.locator("#empty-state")).to_be_hidden()
|
expect(page.locator("#empty-state")).to_be_hidden()
|
||||||
|
|
||||||
# The persisted record re-renders BOTH tool lines, in saved order,
|
# The persisted record re-renders ALL THREE tool lines, in saved
|
||||||
# through the same append helper as the live frames.
|
# order, through the same append helper as the live frames.
|
||||||
restored = page.locator(".msg.brain .tool-call")
|
restored = page.locator(".msg.brain .tool-call")
|
||||||
expect(restored).to_have_count(2)
|
expect(restored).to_have_count(3)
|
||||||
expect(restored.nth(0)).to_contain_text("Listing documents")
|
expect(restored.nth(0)).to_contain_text("Listing documents")
|
||||||
expect(restored.nth(1)).to_contain_text("Reading ")
|
expect(restored.nth(1)).to_contain_text("Listing documents in")
|
||||||
expect(restored.nth(1)).to_contain_text(READ_SP)
|
expect(restored.nth(1)).to_contain_text(READ_SOURCE)
|
||||||
|
expect(restored.nth(2)).to_contain_text("Reading ")
|
||||||
|
expect(restored.nth(2)).to_contain_text(READ_SP)
|
||||||
|
|
||||||
# Answer + the read-document chip are intact (phase-14 restore path).
|
# Answer + the read-document chip are intact (phase-14 restore path).
|
||||||
bubble = page.locator(".msg.brain .bubble").last
|
bubble = page.locator(".msg.brain .bubble").last
|
||||||
|
|||||||
@@ -11,19 +11,26 @@ message contains BOTH ``use your tools`` (``TOOLS_TRIGGER``) and ``read
|
|||||||
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
|
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
|
||||||
carries the ``<tools>`` section of the HIGH prompt; phase 70: the flow
|
carries the ``<tools>`` section of the HIGH prompt; phase 70: the flow
|
||||||
emits the harness-aligned names — ``ls``, then ``read`` on the JOINED
|
emits the harness-aligned names — ``ls``, then ``read`` on the JOINED
|
||||||
combined ``source/path`` of each catalog line):
|
combined ``source/path`` of each file line; phase 94: the drill-down
|
||||||
|
``ls`` — the top level lists sources only, so the flow drills one
|
||||||
|
level into the first source before the first file line exists):
|
||||||
|
|
||||||
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
||||||
``tool_calls`` deltas calling ``ls`` (id ``call_0``);
|
``tool_calls`` deltas calling ``ls`` (id ``call_0``);
|
||||||
2. request 2 (the ``tool``-role catalog result) → ``read`` on the
|
2. request 2 (the top-level source listing in the messages — no file
|
||||||
JOINED combined ``source/path`` of the FIRST catalog line
|
lines yet) → the drill: ``ls`` scoped to the FIRST source of the
|
||||||
(id ``call_1``);
|
listing (id ``call_1``); the seed registers ``Deployments`` first,
|
||||||
3. request 3 (one ``tool``-role read result) → ``read`` on the JOINED
|
and both read documents live in it — so the drill's folder listing
|
||||||
combined ``source/path`` of the SECOND catalog line (id ``call_2``)
|
carries BOTH file lines;
|
||||||
— the pre-phase-45 per-tool budgets would have refused exactly this
|
3. request 3 (a ``tool``-role folder listing with file lines) →
|
||||||
|
``read`` on the JOINED combined ``source/path`` of the FIRST file
|
||||||
|
line (id ``call_2``);
|
||||||
|
4. request 4 (one ``tool``-role read result) → ``read`` on the JOINED
|
||||||
|
combined ``source/path`` of the SECOND file line (id ``call_3``) —
|
||||||
|
the pre-phase-45 per-tool budgets would have refused exactly this
|
||||||
second read (``No reading budget left — answer with what you
|
second read (``No reading budget left — answer with what you
|
||||||
have.``);
|
have.``);
|
||||||
4. request 4 (two read results) → the forced answer, byte-stable: the
|
5. request 5 (two read results) → the forced answer, byte-stable: the
|
||||||
single-read shape quoting the FIRST read result, plus the line
|
single-read shape quoting the FIRST read result, plus the line
|
||||||
``I read <sp1> and <sp2>.`` naming both read paths in read order.
|
``I read <sp1> and <sp2>.`` naming both read paths in read order.
|
||||||
|
|
||||||
@@ -49,10 +56,11 @@ rejection, not the multi-read
|
|||||||
flow this story proves.
|
flow this story proves.
|
||||||
|
|
||||||
Test → story mapping (Playwright Mapping Rule):
|
Test → story mapping (Playwright Mapping Rule):
|
||||||
1. ``test_multi_read_turn`` — the turn streams THREE ``tool`` frames /
|
1. ``test_multi_read_turn`` — the turn streams FOUR ``tool`` frames /
|
||||||
``.tool-call`` lines in order (one list — "is listing documents" —
|
``.tool-call`` lines in order (the top-level list, the drill list —
|
||||||
and two reads — "is reading <source/path>" — the #send-status
|
phase 94 — "is listing documents", and two reads — "is reading
|
||||||
transition recorded deterministically via MutationObserver), then a
|
<source/path>" — the #send-status transition recorded
|
||||||
|
deterministically via MutationObserver), then a
|
||||||
final non-deflected answer containing the mock's byte-stable
|
final non-deflected answer containing the mock's byte-stable
|
||||||
``I read <sp1> and <sp2>.`` line; the round cap (default 10) bounds
|
``I read <sp1> and <sp2>.`` line; the round cap (default 10) bounds
|
||||||
the turn, no budget refusal anywhere.
|
the turn, no budget refusal anywhere.
|
||||||
@@ -84,7 +92,7 @@ from sqlalchemy import select, text
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from app.db import SessionLocal
|
from app.db import SessionLocal
|
||||||
from app.models import Chunk, Document, QueryLog
|
from app.models import Chunk, Document, GitSource, QueryLog
|
||||||
from e2e.auth_helpers import login
|
from e2e.auth_helpers import login
|
||||||
from tests.e2e.mock_llm import embed_text
|
from tests.e2e.mock_llm import embed_text
|
||||||
|
|
||||||
@@ -209,7 +217,23 @@ def _doc(source: str, path: str, title: str, content: str) -> Document:
|
|||||||
|
|
||||||
|
|
||||||
def _seed(db: Session) -> None:
|
def _seed(db: Session) -> None:
|
||||||
"""The three-document KB from the module docstring."""
|
"""The three-document KB from the module docstring.
|
||||||
|
|
||||||
|
Phase 94: the drill-down ``ls`` top level reads the registry —
|
||||||
|
register BOTH sources (TRUNCATEd in ``_reset_db``), ``Deployments``
|
||||||
|
FIRST (registry order is ``(added_at, id)``): the mock's drill
|
||||||
|
(first source of the listing) lands on the folder that carries
|
||||||
|
BOTH file lines. A non-empty table also ignores the operator's
|
||||||
|
``BOR_GIT_SOURCES`` fallback — deterministic.
|
||||||
|
"""
|
||||||
|
# COMMIT between the inserts (not flush): ``added_at`` is
|
||||||
|
# ``server_default now()`` — the transaction timestamp — and the
|
||||||
|
# tie-break is the RANDOM uuid ``id``, so two rows in one
|
||||||
|
# transaction order nondeterministically (the integration
|
||||||
|
# ``registry`` fixture's pattern).
|
||||||
|
db.add(GitSource(url=READ1_SOURCE, kind="local"))
|
||||||
|
db.commit()
|
||||||
|
db.add(GitSource(url=SEED_SOURCE, kind="local"))
|
||||||
md = _doc(SEED_SOURCE, SEED_PATH, "AWS Route 53 Notes", ROUTE53_CONTENT)
|
md = _doc(SEED_SOURCE, SEED_PATH, "AWS Route 53 Notes", ROUTE53_CONTENT)
|
||||||
db.add(md)
|
db.add(md)
|
||||||
db.flush()
|
db.flush()
|
||||||
@@ -240,7 +264,10 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
|
|||||||
"""
|
"""
|
||||||
with SessionLocal() as db:
|
with SessionLocal() as db:
|
||||||
db.execute(
|
db.execute(
|
||||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
text(
|
||||||
|
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||||
|
"kb_overview, git_sources"
|
||||||
|
)
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
if seed is not None:
|
if seed is not None:
|
||||||
@@ -399,13 +426,16 @@ def test_multi_read_turn(
|
|||||||
_submit(page, MULTI_QUESTION)
|
_submit(page, MULTI_QUESTION)
|
||||||
_wait_settled(page)
|
_wait_settled(page)
|
||||||
|
|
||||||
# Wire level: exactly THREE `tool` frames — ls, read #1, read #2
|
# Wire level: exactly FOUR `tool` frames — the top-level ls, the
|
||||||
# (each read's argument is the JOINED combined source/path), in
|
# drill ls scoped to the first source (phase 94), then read #1 and
|
||||||
# order — and all ahead of the first `delta` frame. This third
|
# read #2 (each read's argument is the JOINED combined
|
||||||
# frame is the one the pre-phase-45 read budget refused.
|
# source/path), in order — and all ahead of the first `delta`
|
||||||
|
# frame. The fourth frame is the one the pre-phase-45 read budget
|
||||||
|
# refused.
|
||||||
frames = _frames(page)
|
frames = _frames(page)
|
||||||
assert _tool_frames(frames) == [
|
assert _tool_frames(frames) == [
|
||||||
{"type": "tool", "name": "ls", "argument": None},
|
{"type": "tool", "name": "ls", "argument": None},
|
||||||
|
{"type": "tool", "name": "ls", "argument": READ1_SOURCE},
|
||||||
{"type": "tool", "name": "read", "argument": READ1_SP},
|
{"type": "tool", "name": "read", "argument": READ1_SP},
|
||||||
{"type": "tool", "name": "read", "argument": READ2_SP},
|
{"type": "tool", "name": "read", "argument": READ2_SP},
|
||||||
]
|
]
|
||||||
@@ -439,14 +469,17 @@ def test_multi_read_turn(
|
|||||||
), statuses
|
), statuses
|
||||||
assert i_list < i_read1 < i_read2, statuses
|
assert i_list < i_read1 < i_read2, statuses
|
||||||
|
|
||||||
# Three visible tool lines, in order, above the answer.
|
# Four visible tool lines, in order, above the answer (phase 94:
|
||||||
|
# the drill line is "Listing documents in <source>").
|
||||||
lines = page.locator(".msg.brain .tool-call")
|
lines = page.locator(".msg.brain .tool-call")
|
||||||
expect(lines).to_have_count(3)
|
expect(lines).to_have_count(4)
|
||||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||||
expect(lines.nth(1)).to_contain_text(READ1_SP)
|
expect(lines.nth(1)).to_contain_text(READ1_SOURCE)
|
||||||
expect(lines.nth(2)).to_contain_text("Reading ")
|
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||||
expect(lines.nth(2)).to_contain_text(READ2_SP)
|
expect(lines.nth(2)).to_contain_text(READ1_SP)
|
||||||
|
expect(lines.nth(3)).to_contain_text("Reading ")
|
||||||
|
expect(lines.nth(3)).to_contain_text(READ2_SP)
|
||||||
|
|
||||||
# The final answer is non-deflected, quotes the FIRST read result,
|
# The final answer is non-deflected, quotes the FIRST read result,
|
||||||
# and names BOTH read paths (the mock's byte-stable line).
|
# and names BOTH read paths (the mock's byte-stable line).
|
||||||
@@ -563,19 +596,23 @@ def test_single_tool_flow_regression(
|
|||||||
_submit(page, SINGLE_QUESTION)
|
_submit(page, SINGLE_QUESTION)
|
||||||
_wait_settled(page)
|
_wait_settled(page)
|
||||||
|
|
||||||
# Exactly TWO tool frames — ls then ONE read of the first catalog
|
# Exactly THREE tool frames — the top-level ls, the drill ls
|
||||||
# line (the JOINED combined source/path) — no second read (the
|
# (phase 94), then ONE read of the first file line (the JOINED
|
||||||
# marker carries no multi-read trigger).
|
# combined source/path) — no second read (the marker carries no
|
||||||
|
# multi-read trigger).
|
||||||
frames = _frames(page)
|
frames = _frames(page)
|
||||||
assert _tool_frames(frames) == [
|
assert _tool_frames(frames) == [
|
||||||
{"type": "tool", "name": "ls", "argument": None},
|
{"type": "tool", "name": "ls", "argument": None},
|
||||||
|
{"type": "tool", "name": "ls", "argument": READ1_SOURCE},
|
||||||
{"type": "tool", "name": "read", "argument": READ1_SP},
|
{"type": "tool", "name": "read", "argument": READ1_SP},
|
||||||
]
|
]
|
||||||
lines = page.locator(".msg.brain .tool-call")
|
lines = page.locator(".msg.brain .tool-call")
|
||||||
expect(lines).to_have_count(2)
|
expect(lines).to_have_count(3)
|
||||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||||
expect(lines.nth(1)).to_contain_text(READ1_SP)
|
expect(lines.nth(1)).to_contain_text(READ1_SOURCE)
|
||||||
|
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||||
|
expect(lines.nth(2)).to_contain_text(READ1_SP)
|
||||||
|
|
||||||
# The single-read answer shape: quotes the read document; it does
|
# The single-read answer shape: quotes the read document; it does
|
||||||
# NOT carry the multi-read both-named line (READ2 was never read).
|
# NOT carry the multi-read both-named line (READ2 was never read).
|
||||||
|
|||||||
@@ -19,27 +19,33 @@ phase-64 default 0.15 is far below the 5 s tool-line threshold):
|
|||||||
|
|
||||||
NOTE the prelude: the chat flow's QUESTION EMBEDDING also travels
|
NOTE the prelude: the chat flow's QUESTION EMBEDDING also travels
|
||||||
through the proxy (one 6 s sleep) before the agent loop starts, so
|
through the proxy (one 6 s sleep) before the agent loop starts, so
|
||||||
the wall-clock timeline of one turn is:
|
the wall-clock timeline of one turn is (phase 94: the drill-down
|
||||||
|
``ls`` adds a drill step — the top level lists sources only, so the
|
||||||
|
flow drills one level before the first file line exists):
|
||||||
|
|
||||||
1. t≈12 s — the first SSE ``tool`` frame ("🔎 Listing documents"): the
|
1. t≈12 s — the first SSE ``tool`` frame ("🔎 Listing documents"): the
|
||||||
6 s embedding sleep + request 1's 6 s sleep (``tools`` offered, no
|
6 s embedding sleep + request 1's 6 s sleep (``tools`` offered, no
|
||||||
tool results yet — streams ONLY the ``ls`` ``tool_calls`` delta,
|
tool results yet — streams ONLY the ``ls`` ``tool_calls`` delta,
|
||||||
id ``call_0``) + the mock's ~0.2 s tool-call stream;
|
id ``call_0``) + the mock's ~0.2 s tool-call stream;
|
||||||
2. t≈19 s — the second ``tool`` frame ("📄 Reading
|
2. t≈19 s — the second ``tool`` frame ("🔎 Listing documents in
|
||||||
Deployments/example-record-file.json"): request 2's 6 s sleep (a
|
Deployments"): request 2's 6 s sleep (the top-level source listing
|
||||||
``tool``-role catalog result → streams a ``read`` ``tool_calls``
|
in the messages — no file lines yet → streams the drill ``ls``
|
||||||
delta on the JOINED combined ``source/path`` of the FIRST catalog
|
scoped to the first source, id ``call_1``);
|
||||||
line, id ``call_1``);
|
3. t≈25 s — the third ``tool`` frame ("📄 Reading
|
||||||
3. t≈25 s — the content answer ``Read <source/path>. <first 80 chars
|
Deployments/example-record-file.json"): request 3's 6 s sleep (a
|
||||||
of the read document's content>``: request 3's 6 s sleep (a
|
``tool``-role folder listing with file lines → streams a ``read``
|
||||||
|
``tool_calls`` delta on the JOINED combined ``source/path`` of the
|
||||||
|
FIRST file line, id ``call_2``);
|
||||||
|
4. t≈31 s — the content answer ``Read <source/path>. <first 80 chars
|
||||||
|
of the read document's content>``: request 4's 6 s sleep (a
|
||||||
``tool``-role read result) → the first answer ``delta``.
|
``tool``-role read result) → the first answer ``delta``.
|
||||||
|
|
||||||
So each post-tool-frame gap (≈6.3 s — the 6 s sleep + the short
|
So each post-tool-frame gap (≈6.3 s — the 6 s sleep + the short
|
||||||
tool-call stream) is PAST the 5 s tool-line threshold
|
tool-call stream) is PAST the 5 s tool-line threshold
|
||||||
(``TOOL_LINE_ELAPSED_AFTER_MS = 5_000``), and the 10 s pre-token
|
(``TOOL_LINE_ELAPSED_AFTER_MS = 5_000``), and the 10 s pre-token
|
||||||
typing hint (the existing ``startThinkingClock`` gate, ticking from
|
typing hint (the existing ``startThinkingClock`` gate, ticking from
|
||||||
t≈0) is visible throughout both post-tool gaps — everything before the
|
t≈0) is visible throughout all post-tool gaps — everything before the
|
||||||
answer's first delta (the whole turn runs ≈25 s + overhead, acceptable
|
answer's first delta (the whole turn runs ≈31 s + overhead, acceptable
|
||||||
for an isolated story suite). The four tests pin: (1) the latest tool
|
for an isolated story suite). The four tests pin: (1) the latest tool
|
||||||
line's ticking "(Ns)" suffix, (2) the typing indicator's visible "Ns"
|
line's ticking "(Ns)" suffix, (2) the typing indicator's visible "Ns"
|
||||||
hint (+ the kept aria channel), (3) BOTH settling the instant the
|
hint (+ the kept aria channel), (3) BOTH settling the instant the
|
||||||
@@ -55,15 +61,20 @@ task that writes the visible span) — so the two channels read
|
|||||||
polls until they align (the deterministic same-moment pin).
|
polls until they align (the deterministic same-moment pin).
|
||||||
|
|
||||||
**KB fixture** — byte-identical to the phase-37 suite
|
**KB fixture** — byte-identical to the phase-37 suite
|
||||||
(``tests/e2e/test_agent_document_tools.py``): ``Homelab/aws-route53.md``
|
(``tests/e2e/test_agent_document_tools.py``), plus the phase-94
|
||||||
|
registry rows: both sources are registered in ``git_sources``
|
||||||
|
(``Deployments`` FIRST — registry order ``(added_at, id)``), so the
|
||||||
|
mock's drill (first source of the top-level listing) lands on the
|
||||||
|
folder that carries the file lines. ``Homelab/aws-route53.md``
|
||||||
carries one chunk embedded with the mock's own bag-of-words vector —
|
carries one chunk embedded with the mock's own bag-of-words vector —
|
||||||
the marker question (phase 37's exact question, which carries the
|
the marker question (phase 37's exact question, which carries the
|
||||||
trigger phrase) cosines ≈0.69 against it, well past the E2E 0.30
|
trigger phrase) cosines ≈0.69 against it, well past the E2E 0.30
|
||||||
threshold, so the turn is grounded and the HIGH prompt carries the
|
threshold, so the turn is grounded and the HIGH prompt carries the
|
||||||
``<tools>`` section; ``Deployments/example-record-file.json`` is
|
``<tools>`` section; ``Deployments/example-record-file.json`` is
|
||||||
indexed WITHOUT chunks, and its ``(source, path)`` sorts FIRST in the
|
indexed WITHOUT chunks, and its ``(source, path)`` sorts FIRST in the
|
||||||
catalog (``Deployments`` < ``Homelab``) — exactly the line the mock's
|
folder listing's file lines (``Deployments`` < ``Homelab``) — exactly
|
||||||
second request reads, so the answer is byte-stable.
|
the line the mock's read request targets, so the answer is
|
||||||
|
byte-stable.
|
||||||
|
|
||||||
Test → story mapping (Playwright Mapping Rule):
|
Test → story mapping (Playwright Mapping Rule):
|
||||||
1. ``test_tool_line_shows_ticking_elapsed``
|
1. ``test_tool_line_shows_ticking_elapsed``
|
||||||
@@ -92,7 +103,7 @@ from sqlalchemy.orm import Session
|
|||||||
|
|
||||||
from app.config import Settings as _Settings
|
from app.config import Settings as _Settings
|
||||||
from app.db import SessionLocal
|
from app.db import SessionLocal
|
||||||
from app.models import Chunk, Document
|
from app.models import Chunk, Document, GitSource
|
||||||
from e2e.auth_helpers import login
|
from e2e.auth_helpers import login
|
||||||
from e2e.conftest import (
|
from e2e.conftest import (
|
||||||
ADMIN_PASSWORD,
|
ADMIN_PASSWORD,
|
||||||
@@ -211,8 +222,10 @@ SUFFIX_TIMEOUT_MS = 12_000
|
|||||||
#: the indicator is removed at the first delta (≈25 s): 20 s from the
|
#: the indicator is removed at the first delta (≈25 s): 20 s from the
|
||||||
#: check start cannot flake.
|
#: check start cannot flake.
|
||||||
TYPING_HINT_TIMEOUT_MS = 20_000
|
TYPING_HINT_TIMEOUT_MS = 20_000
|
||||||
#: The answer settles at ≈19 s from submit; every settle wait here starts
|
#: The answer settles at ≈31 s from submit (phase 94: the drill step
|
||||||
#: ≥12 s in — 30 s keeps ≥2× headroom on the remaining window.
|
#: adds a third tool round); every settle wait here starts ≈17 s in
|
||||||
|
#: (after the first line's suffix) — 30 s keeps ≥2× headroom on the
|
||||||
|
#: remaining window.
|
||||||
SETTLE_TIMEOUT_MS = 30_000
|
SETTLE_TIMEOUT_MS = 30_000
|
||||||
#: The restore is a synchronous boot re-render — 15 s is ample.
|
#: The restore is a synchronous boot re-render — 15 s is ample.
|
||||||
RESTORE_TIMEOUT_MS = 15_000
|
RESTORE_TIMEOUT_MS = 15_000
|
||||||
@@ -360,7 +373,16 @@ def app_url(app_server: str) -> str:
|
|||||||
|
|
||||||
def _seed(db: Session) -> None:
|
def _seed(db: Session) -> None:
|
||||||
"""The phase-37 two-document pair, byte-identical (see the module
|
"""The phase-37 two-document pair, byte-identical (see the module
|
||||||
docstring)."""
|
docstring), plus the phase-94 registry rows: both sources
|
||||||
|
registered, ``Deployments`` FIRST — the mock's drill (first
|
||||||
|
source of the top-level listing) lands on the JSON file."""
|
||||||
|
# COMMIT between the inserts (not flush): ``added_at`` is
|
||||||
|
# ``server_default now()`` — the transaction timestamp — and the
|
||||||
|
# tie-break is the random uuid ``id``, so one-transaction rows order
|
||||||
|
# nondeterministically.
|
||||||
|
db.add(GitSource(url=READ_SOURCE, kind="local"))
|
||||||
|
db.commit()
|
||||||
|
db.add(GitSource(url=SEED_SOURCE, kind="local"))
|
||||||
md = Document(
|
md = Document(
|
||||||
source=SEED_SOURCE,
|
source=SEED_SOURCE,
|
||||||
path=SEED_PATH,
|
path=SEED_PATH,
|
||||||
@@ -408,7 +430,7 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
|
|||||||
db.execute(
|
db.execute(
|
||||||
text(
|
text(
|
||||||
"TRUNCATE chunks, documents, query_log, "
|
"TRUNCATE chunks, documents, query_log, "
|
||||||
"steering_notes, kb_overview, saved_chats"
|
"steering_notes, kb_overview, saved_chats, git_sources"
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
@@ -514,7 +536,7 @@ def _wait_settled(page: Page) -> None:
|
|||||||
|
|
||||||
#: The FIRST tool line's suffix, pinned to the line ("first .tool-call
|
#: The FIRST tool line's suffix, pinned to the line ("first .tool-call
|
||||||
#: child of the .tool-calls container" — the "Listing documents" line;
|
#: child of the .tool-calls container" — the "Listing documents" line;
|
||||||
#: the read frame's second line arms its OWN clock 5 s later and is
|
#: the drill frame's second line arms its OWN clock 5 s later and is
|
||||||
#: never the target here).
|
#: never the target here).
|
||||||
FIRST_LINE_SUFFIX = "#messages .tool-calls .tool-call:first-child .tool-elapsed"
|
FIRST_LINE_SUFFIX = "#messages .tool-calls .tool-call:first-child .tool-elapsed"
|
||||||
|
|
||||||
@@ -611,8 +633,8 @@ def test_typing_indicator_shows_visible_elapsed(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# The clock ticks while the gap holds: ≥1.5 s later the value is
|
# The clock ticks while the gap holds: ≥1.5 s later the value is
|
||||||
# strictly greater (the read frame is ≥6 s away; the first delta
|
# strictly greater (the drill frame is ≥6 s away; the first delta
|
||||||
# — which removes the whole indicator — is ≈25 s from submit).
|
# — which removes the whole indicator — is ≈31 s from submit).
|
||||||
page.wait_for_timeout(int(SAMPLE_GAP_S * 1000))
|
page.wait_for_timeout(int(SAMPLE_GAP_S * 1000))
|
||||||
v2 = _elapsed_value(
|
v2 = _elapsed_value(
|
||||||
page.locator("#typing-indicator .typing-elapsed").first.text_content(),
|
page.locator("#typing-indicator .typing-elapsed").first.text_content(),
|
||||||
@@ -644,7 +666,7 @@ def test_indicators_settle_when_the_answer_arrives(
|
|||||||
# First the ticking state (the test-1 wait, reused)…
|
# First the ticking state (the test-1 wait, reused)…
|
||||||
expect(page.locator(FIRST_LINE_SUFFIX)).to_be_visible(timeout=SUFFIX_TIMEOUT_MS)
|
expect(page.locator(FIRST_LINE_SUFFIX)).to_be_visible(timeout=SUFFIX_TIMEOUT_MS)
|
||||||
|
|
||||||
# …then the answer arrives (≈25 s from submit — see the module
|
# …then the answer arrives (≈31 s from submit — see the module
|
||||||
# docstring's timeline) and settles.
|
# docstring's timeline) and settles.
|
||||||
_wait_settled(page)
|
_wait_settled(page)
|
||||||
|
|
||||||
@@ -661,13 +683,16 @@ def test_indicators_settle_when_the_answer_arrives(
|
|||||||
expect(page.locator("#messages .tool-elapsed")).to_have_count(0)
|
expect(page.locator("#messages .tool-elapsed")).to_have_count(0)
|
||||||
expect(page.locator("#typing-indicator")).to_have_count(0)
|
expect(page.locator("#typing-indicator")).to_have_count(0)
|
||||||
|
|
||||||
# The tool lines remain — the permanent record, both present with
|
# The tool lines remain — the permanent record, all three present
|
||||||
# their pinned text (phase 37's pattern).
|
# with their pinned text (phase 37's pattern + the phase-94 drill
|
||||||
|
# line).
|
||||||
lines = page.locator("#messages .tool-call")
|
lines = page.locator("#messages .tool-call")
|
||||||
expect(lines).to_have_count(2)
|
expect(lines).to_have_count(3)
|
||||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||||
expect(lines.nth(1)).to_contain_text(READ_SP)
|
expect(lines.nth(1)).to_contain_text(READ_SOURCE)
|
||||||
|
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||||
|
expect(lines.nth(2)).to_contain_text(READ_SP)
|
||||||
|
|
||||||
# The answer bubble is complete (the mock's deterministic quote).
|
# The answer bubble is complete (the mock's deterministic quote).
|
||||||
bubble = page.locator(".msg.brain .bubble").last
|
bubble = page.locator(".msg.brain .bubble").last
|
||||||
@@ -705,12 +730,14 @@ def test_restored_turn_has_no_timer(
|
|||||||
expect(page.locator("#empty-state")).to_be_hidden(timeout=RESTORE_TIMEOUT_MS)
|
expect(page.locator("#empty-state")).to_be_hidden(timeout=RESTORE_TIMEOUT_MS)
|
||||||
|
|
||||||
# The persisted record re-renders the tool lines, in saved order
|
# The persisted record re-renders the tool lines, in saved order
|
||||||
# (the phase-37 reload pin, mirrored)…
|
# (the phase-37 reload pin, mirrored + the phase-94 drill line)…”
|
||||||
restored = page.locator("#messages .tool-call")
|
restored = page.locator("#messages .tool-call")
|
||||||
expect(restored).to_have_count(2, timeout=RESTORE_TIMEOUT_MS)
|
expect(restored).to_have_count(3, timeout=RESTORE_TIMEOUT_MS)
|
||||||
expect(restored.nth(0)).to_contain_text("Listing documents")
|
expect(restored.nth(0)).to_contain_text("Listing documents")
|
||||||
expect(restored.nth(1)).to_contain_text("Reading ")
|
expect(restored.nth(1)).to_contain_text("Listing documents in")
|
||||||
expect(restored.nth(1)).to_contain_text(READ_SP)
|
expect(restored.nth(1)).to_contain_text(READ_SOURCE)
|
||||||
|
expect(restored.nth(2)).to_contain_text("Reading ")
|
||||||
|
expect(restored.nth(2)).to_contain_text(READ_SP)
|
||||||
|
|
||||||
# …with NO timer (A6 — the restore never arms the clock) and the
|
# …with NO timer (A6 — the restore never arms the clock) and the
|
||||||
# page otherwise settled.
|
# page otherwise settled.
|
||||||
|
|||||||
@@ -13,9 +13,13 @@ deterministic marker flows in ``tests/e2e/mock_llm.py`` (phase 70: the
|
|||||||
flows emit the NEW names with the NEW argument shapes):
|
flows emit the NEW names with the NEW argument shapes):
|
||||||
|
|
||||||
* the READ flow (``use your tools`` (``TOOLS_TRIGGER``) + the HIGH
|
* the READ flow (``use your tools`` (``TOOLS_TRIGGER``) + the HIGH
|
||||||
prompt's ``<tools>`` section): ``ls`` (id ``call_0``, no arguments) →
|
prompt's ``<tools>`` section; phase 94: the drill-down ``ls`` — the
|
||||||
``read`` on the JOINED combined ``source/path`` of the first catalog
|
top level lists sources only, so the flow drills one level before the
|
||||||
line (id ``call_1``) → the ``Read <source/path>. <quote>`` answer;
|
first file line exists): ``ls`` (id ``call_0``, no arguments) → the
|
||||||
|
drill ``ls`` scoped to the first source of the listing
|
||||||
|
(id ``call_1``) → ``read`` on the JOINED combined ``source/path`` of
|
||||||
|
the first file line (id ``call_2``) → the ``Read <source/path>.
|
||||||
|
<quote>`` answer;
|
||||||
* the SEARCH flow (``search your documents`` (``SEARCH_TRIGGER``) + the
|
* the SEARCH flow (``search your documents`` (``SEARCH_TRIGGER``) + the
|
||||||
``<tools>`` section): ``grep`` with ``{"pattern": SEARCH_PATTERN}``
|
``<tools>`` section): ``grep`` with ``{"pattern": SEARCH_PATTERN}``
|
||||||
(id ``call_0``) → the ``Found <matched line>`` answer.
|
(id ``call_0``) → the ``Found <matched line>`` answer.
|
||||||
@@ -41,13 +45,14 @@ KB fixtures:
|
|||||||
|
|
||||||
Test → phase mapping (Playwright Mapping Rule):
|
Test → phase mapping (Playwright Mapping Rule):
|
||||||
1. ``test_read_flow_lines_answer_sources_no_raw_markup`` — the
|
1. ``test_read_flow_lines_answer_sources_no_raw_markup`` — the
|
||||||
grounded READ turn: the UI shows the ``ls`` line (unscoped "🔎
|
grounded READ turn: the UI shows the unscoped ``ls`` line ("🔎
|
||||||
Listing documents", no argument) then the "📄 Reading <source/path>"
|
Listing documents", no argument), the drill line ("🔎 Listing
|
||||||
line with the combined path in a ``<code>`` element, the answer
|
documents in <source>" — phase 94, the source in a ``<code>``
|
||||||
streams and quotes the read document, the done-state sources
|
element), then the "📄 Reading <source/path>" line with the
|
||||||
include the read document, and NO raw tool markup (``<|…|>``,
|
combined path in a ``<code>`` element, the answer streams and quotes
|
||||||
``tool_call``) appears anywhere in the DOM — the live incident this
|
the read document, the done-state sources include the read document,
|
||||||
phase fixes.
|
and NO raw tool markup (``<|…|>``, ``tool_call``) appears anywhere
|
||||||
|
in the DOM — the live incident this phase fixes.
|
||||||
2. ``test_grep_flow_line_then_answer`` — the grounded SEARCH turn: the
|
2. ``test_grep_flow_line_then_answer`` — the grounded SEARCH turn: the
|
||||||
"🔎 Searching for <pattern>" line (sentinel in ``<code>``) then the
|
"🔎 Searching for <pattern>" line (sentinel in ``<code>``) then the
|
||||||
matched-line answer.
|
matched-line answer.
|
||||||
@@ -74,7 +79,7 @@ from sqlalchemy.orm import Session
|
|||||||
|
|
||||||
from app.config import Settings
|
from app.config import Settings
|
||||||
from app.db import SessionLocal
|
from app.db import SessionLocal
|
||||||
from app.models import Chunk, Document
|
from app.models import Chunk, Document, GitSource
|
||||||
from app.rag.importer import ImportSummary, import_sources
|
from app.rag.importer import ImportSummary, import_sources
|
||||||
from app.rag.llm import LLMClient
|
from app.rag.llm import LLMClient
|
||||||
from e2e.auth_helpers import login
|
from e2e.auth_helpers import login
|
||||||
@@ -156,7 +161,22 @@ READ_ANSWER_QUOTE = RECORD_FILE_CONTENT[:80]
|
|||||||
|
|
||||||
|
|
||||||
def _seed_read_pair(db: Session) -> None:
|
def _seed_read_pair(db: Session) -> None:
|
||||||
"""The two-document READ-flow KB (see the module docstring)."""
|
"""The two-document READ-flow KB (see the module docstring).
|
||||||
|
|
||||||
|
Phase 94: the drill-down ``ls`` top level reads the registry —
|
||||||
|
register BOTH sources (TRUNCATEd in ``_reset_db_read_pair``),
|
||||||
|
``Deployments`` FIRST (registry order ``(added_at, id)``): the
|
||||||
|
mock's drill (first source of the listing) lands on the JSON file
|
||||||
|
— the read the assertions expect. A non-empty table also ignores
|
||||||
|
the operator's ``BOR_GIT_SOURCES`` fallback — deterministic.
|
||||||
|
"""
|
||||||
|
# COMMIT between the inserts (not flush): ``added_at`` is
|
||||||
|
# ``server_default now()`` — the transaction timestamp — and the
|
||||||
|
# tie-break is the random uuid ``id``, so one-transaction rows order
|
||||||
|
# nondeterministically.
|
||||||
|
db.add(GitSource(url=READ_SOURCE, kind="local"))
|
||||||
|
db.commit()
|
||||||
|
db.add(GitSource(url=SEED_SOURCE, kind="local"))
|
||||||
md = Document(
|
md = Document(
|
||||||
source=SEED_SOURCE,
|
source=SEED_SOURCE,
|
||||||
path=SEED_PATH,
|
path=SEED_PATH,
|
||||||
@@ -267,7 +287,10 @@ def _reset_db_read_pair() -> None:
|
|||||||
answers."""
|
answers."""
|
||||||
with SessionLocal() as db:
|
with SessionLocal() as db:
|
||||||
db.execute(
|
db.execute(
|
||||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
text(
|
||||||
|
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||||
|
"kb_overview, git_sources"
|
||||||
|
)
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
_seed_read_pair(db)
|
_seed_read_pair(db)
|
||||||
@@ -386,15 +409,18 @@ def test_read_flow_lines_answer_sources_no_raw_markup(
|
|||||||
_submit(page, READ_QUESTION)
|
_submit(page, READ_QUESTION)
|
||||||
_wait_settled(page)
|
_wait_settled(page)
|
||||||
|
|
||||||
# The UI shows the ls line (UNSCOPED — no argument, no <code>) then
|
# The UI shows the ls line (UNSCOPED — no argument, no <code>),
|
||||||
|
# the drill line (phase 94 — the source in a <code> element), then
|
||||||
# the "📄 Reading <source/path>" line with the COMBINED path in a
|
# the "📄 Reading <source/path>" line with the COMBINED path in a
|
||||||
# <code> element (the path is data, never markup).
|
# <code> element (the path is data, never markup).
|
||||||
lines = page.locator(".msg.brain .tool-call")
|
lines = page.locator(".msg.brain .tool-call")
|
||||||
expect(lines).to_have_count(2)
|
expect(lines).to_have_count(3)
|
||||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||||
expect(lines.nth(0).locator("code")).to_have_count(0)
|
expect(lines.nth(0).locator("code")).to_have_count(0)
|
||||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||||
expect(lines.nth(1).locator("code")).to_have_text(READ_SP)
|
expect(lines.nth(1).locator("code")).to_have_text(READ_SOURCE)
|
||||||
|
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||||
|
expect(lines.nth(2).locator("code")).to_have_text(READ_SP)
|
||||||
|
|
||||||
# The answer streamed and quotes the read document (the mock's
|
# The answer streamed and quotes the read document (the mock's
|
||||||
# deterministic echo: "Read <source/path>. <first 80 chars>").
|
# deterministic echo: "Read <source/path>. <first 80 chars>").
|
||||||
@@ -402,12 +428,14 @@ def test_read_flow_lines_answer_sources_no_raw_markup(
|
|||||||
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
|
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
|
||||||
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
||||||
|
|
||||||
# Wire level: ls then read — the phase-70 argument rule (ls
|
# Wire level: ls, the drill ls scoped to the first source (phase
|
||||||
# unscoped → null; read → the combined path as passed) — ahead of
|
# 94), then read — the phase-70 argument rule (ls unscoped → null;
|
||||||
# the first delta.
|
# scoped ls → the source name; read → the combined path as passed)
|
||||||
|
# — ahead of the first delta.
|
||||||
frames = _drain_frames(page)
|
frames = _drain_frames(page)
|
||||||
assert _tool_frames(frames) == [
|
assert _tool_frames(frames) == [
|
||||||
{"type": "tool", "name": "ls", "argument": None},
|
{"type": "tool", "name": "ls", "argument": None},
|
||||||
|
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
|
||||||
{"type": "tool", "name": "read", "argument": READ_SP},
|
{"type": "tool", "name": "read", "argument": READ_SP},
|
||||||
]
|
]
|
||||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||||
@@ -508,10 +536,12 @@ def test_wire_argument_rule_across_both_flows(
|
|||||||
read_tools = _tool_frames(read_frames)
|
read_tools = _tool_frames(read_frames)
|
||||||
search_tools = _tool_frames(search_frames)
|
search_tools = _tool_frames(search_frames)
|
||||||
# The ordered, combined tool-frame sequence across both flows: the
|
# The ordered, combined tool-frame sequence across both flows: the
|
||||||
# argument rule end-to-end — read → the combined path as passed,
|
# argument rule end-to-end — the drill ls (phase 94) → the source
|
||||||
# grep → the pattern, ls → null when unscoped.
|
# name, read → the combined path as passed, grep → the pattern,
|
||||||
|
# ls → null when unscoped.
|
||||||
assert read_tools + search_tools == [
|
assert read_tools + search_tools == [
|
||||||
{"type": "tool", "name": "ls", "argument": None},
|
{"type": "tool", "name": "ls", "argument": None},
|
||||||
|
{"type": "tool", "name": "ls", "argument": READ_SOURCE},
|
||||||
{"type": "tool", "name": "read", "argument": READ_SP},
|
{"type": "tool", "name": "read", "argument": READ_SP},
|
||||||
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN},
|
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN},
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -0,0 +1,772 @@
|
|||||||
|
"""Phase 94 task 04 E2E (Playwright, mock-only): the ``ls`` drill-down tree.
|
||||||
|
|
||||||
|
The dedicated story suite for ``94_ls_tree_drilldown`` (owner TODO.md L4):
|
||||||
|
the LLM drills ``ls()`` → ``ls(source)`` → ``ls(source/folder)`` →
|
||||||
|
``read(source/file)`` through the real UI, the sync-time folder summaries
|
||||||
|
(must exist after a changed sync under the deterministic mock's
|
||||||
|
``FOLDER_SUMMARY_MODE`` branch) show up in the ``ls`` output, and the
|
||||||
|
50-line cap holds on a wide folder.
|
||||||
|
|
||||||
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||||
|
|
||||||
|
uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov
|
||||||
|
|
||||||
|
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real ``turbo``
|
||||||
|
does whatever it does with the tools, while this story's gate is the
|
||||||
|
deterministic SCRIPTED drill-down flow in ``tests/e2e/mock_llm.py``
|
||||||
|
(``DRILL_TRIGGER``: user message contains ``drill down the tree`` —
|
||||||
|
the question carries its own tool call after the colon,
|
||||||
|
``drill down the tree: ls [target]`` / ``drill down the tree: read
|
||||||
|
source/path`` — **and** the system prompt carries the ``<tools>``
|
||||||
|
section of the HIGH prompt). The mock echoes the received tool result
|
||||||
|
into its final answer (the house scripted-turn way of asserting on tool
|
||||||
|
results — the mock is the only E2E lens on the LLM's context), so every
|
||||||
|
tree-level assertion below lands on the rendered answer; the DOM
|
||||||
|
assertions cover the tool lines + the answer.
|
||||||
|
|
||||||
|
KB fixture — a host temp dir tree (``tmp_path_factory``; the app runs on
|
||||||
|
the same host) with TWO registered local sources (the
|
||||||
|
``test_local_directory_sources.py`` registration + real-Sync pattern —
|
||||||
|
registration through the authenticated API, the real in-process
|
||||||
|
``POST /api/sync`` pipeline; no git anywhere):
|
||||||
|
|
||||||
|
* ``alpha/`` — ``root-note.md`` at the source root, ``one/`` (2 docs),
|
||||||
|
``two/`` (2 docs — the read target lives here) and ``wide/`` (51 tiny
|
||||||
|
files — the 50-line cap subject);
|
||||||
|
* ``beta/`` — ``gamma/`` (2 docs).
|
||||||
|
|
||||||
|
Every fixture doc carries the words ``drill down the tree`` in its body,
|
||||||
|
so every scripted question (which contains the trigger phrase) FTS-matches
|
||||||
|
at least one chunk — the honesty gate is HIGH for all turns regardless
|
||||||
|
of the mock's cosine distribution, and the ``<tools>`` section is present
|
||||||
|
(the flow's precondition).
|
||||||
|
|
||||||
|
The mock's canned ``FOLDER_SUMMARY_MODE`` branch (phase 94 task 01)
|
||||||
|
stores, per ≥ 2-doc folder, the deterministic one-liner
|
||||||
|
``Fixture folder summary for <source>[/<folder>].`` — the ``synced_kb``
|
||||||
|
module fixture pins those exact rows in ``folder_summaries`` after the
|
||||||
|
sync, and the drill answers assert on them in the ``ls`` output.
|
||||||
|
|
||||||
|
Test → story mapping (Playwright Mapping Rule):
|
||||||
|
1. ``test_drill_down_sources_folders_files_and_read`` — the four
|
||||||
|
scripted turns: ``ls()`` (the ``🔎 Listing documents`` line + the
|
||||||
|
top-level shape — one ``— N documents`` line per source + the
|
||||||
|
canned source-root summaries), ``ls(alpha)`` (folder lines with
|
||||||
|
their summaries + the root file line), ``ls(alpha/two)`` (the exact
|
||||||
|
``source: X | path: Y | title: Z`` file lines), and the grounded
|
||||||
|
``read(alpha/two/two-a.md)`` (the ``📄 Reading …`` line + the answer
|
||||||
|
citing the document, the phase-37 assertion pattern). The read
|
||||||
|
target is DELIBERATELY a top-2 retrieval document for its question
|
||||||
|
(the question names the file's path, so the file self-matches the
|
||||||
|
hybrid gate deterministically): the agent's phase-72 dedupe returns
|
||||||
|
``ALREADY_IN_CONTEXT`` (a refusal — counts in nothing), and the
|
||||||
|
mock answers FROM THE ``<documents>`` PROMPT with the same citation
|
||||||
|
shape (``Already in context: Read <sp>. <first 80 chars>``) — the
|
||||||
|
document text reached the model either way, and the prefix pins
|
||||||
|
that the dedupe notice itself reached it.
|
||||||
|
2. ``test_wide_folder_holds_the_fifty_line_cap`` — ``ls(alpha/wide)``
|
||||||
|
on the 51-file folder: the mock's echo carries exactly 50 file lines
|
||||||
|
+ the ``…and 1 more documents in this folder — use grep (pattern)…``
|
||||||
|
note; the 51st file never reaches the model.
|
||||||
|
3. ``test_not_a_folder_teaching_and_scripted_recovery`` — the mock calls
|
||||||
|
``ls(alpha/nope)``; the NOT-A-FOLDER teaching line (the argument
|
||||||
|
echoed, the parent's subfolders listed) is visible to the model —
|
||||||
|
the mock's scripted recovery branch keys on receiving it — and the
|
||||||
|
scripted ``ls(alpha)`` recovery lands (the answer is the parent's
|
||||||
|
listing; the loop ends in one refusal + one correction, not at the
|
||||||
|
round cap).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pytest
|
||||||
|
from playwright.sync_api import Locator, Page, expect
|
||||||
|
from sqlalchemy import select, text
|
||||||
|
|
||||||
|
from app.config import Settings as _Settings
|
||||||
|
from app.db import SessionLocal
|
||||||
|
from app.models import FolderSummary, QueryLog
|
||||||
|
from e2e.auth_helpers import login
|
||||||
|
from e2e.conftest import (
|
||||||
|
ADMIN_PASSWORD,
|
||||||
|
SESSION_SECRET,
|
||||||
|
USE_REAL_LLM,
|
||||||
|
_wait_http,
|
||||||
|
)
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
|
||||||
|
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||||
|
# port in a combined run — this module app binds its own port instead
|
||||||
|
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||||
|
# server). Env-overridable.
|
||||||
|
APP_PORT = int(os.environ.get("E2E_APP_PORT_LSTREE", "8136"))
|
||||||
|
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# Fixture documents (deterministic, token-controlled)
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
ALPHA = "alpha"
|
||||||
|
BETA = "beta"
|
||||||
|
|
||||||
|
ROOT_NOTE = "root-note.md"
|
||||||
|
TWO_A = "two/two-a.md"
|
||||||
|
TWO_B = "two/two-b.md"
|
||||||
|
READ_SP = f"{ALPHA}/{TWO_A}"
|
||||||
|
|
||||||
|
ALPHA_COUNT = 56 # 1 root note + 2 one/ + 2 two/ + 51 wide/
|
||||||
|
BETA_COUNT = 2
|
||||||
|
TOTAL_DOCS = ALPHA_COUNT + BETA_COUNT
|
||||||
|
WIDE_COUNT = 51
|
||||||
|
LS_MAX_FILE_LINES = 50 # app.rag.agent.LS_MAX_FILE_LINES — the cap under test
|
||||||
|
|
||||||
|
#: Every fixture body carries ``drill down the tree`` (the trigger
|
||||||
|
#: phrase's words): every scripted question FTS-matches at least one
|
||||||
|
#: chunk → HIGH gate → the ``<tools>`` section the flow keys on.
|
||||||
|
DRILL_LEAD = "The drill down the tree fixture note"
|
||||||
|
|
||||||
|
|
||||||
|
def _md(title: str, body: str) -> str:
|
||||||
|
return f"# {title}\n\n{body}\n"
|
||||||
|
|
||||||
|
|
||||||
|
#: The read target — its FIRST line is ≥ 80 chars, so the mock's
|
||||||
|
#: first-80-chars quote (the phase-37 single-read shape) is newline-free
|
||||||
|
#: and the rendered-text assertion matches it verbatim. Pinned by the
|
||||||
|
#: assert below.
|
||||||
|
TWO_A_TITLE = (
|
||||||
|
"Alpha Two A — the drill-down read target for the alpha two folder "
|
||||||
|
"listing turn in the brain of reese fixture"
|
||||||
|
)
|
||||||
|
TWO_A_CONTENT = _md(
|
||||||
|
TWO_A_TITLE,
|
||||||
|
f"{DRILL_LEAD} for alpha two: this document covers topic A of the "
|
||||||
|
"alpha source tree; it is the file the scripted read turn opens "
|
||||||
|
"from the alpha/two listing.",
|
||||||
|
)
|
||||||
|
assert "\n" not in TWO_A_CONTENT[:80] # the quote must stay one line
|
||||||
|
|
||||||
|
#: The sync-time folder summaries the mock's canned ``FOLDER_SUMMARY_MODE``
|
||||||
|
#: branch stores (task 01's byte-stable template), in
|
||||||
|
#: ``(source, folder_path)`` order: one row per ≥ 2-doc folder (the
|
||||||
|
#: recursive-subtree rule) — the ``""`` rows are the source roots.
|
||||||
|
EXPECTED_SUMMARIES: list[tuple[str, str, str]] = [
|
||||||
|
(ALPHA, "", f"Fixture folder summary for {ALPHA}."),
|
||||||
|
(ALPHA, "one", f"Fixture folder summary for {ALPHA}/one."),
|
||||||
|
(ALPHA, "two", f"Fixture folder summary for {ALPHA}/two."),
|
||||||
|
(ALPHA, "wide", f"Fixture folder summary for {ALPHA}/wide."),
|
||||||
|
(BETA, "", f"Fixture folder summary for {BETA}."),
|
||||||
|
(BETA, "gamma", f"Fixture folder summary for {BETA}/gamma."),
|
||||||
|
]
|
||||||
|
assert [
|
||||||
|
(source, folder) for source, folder, _s in EXPECTED_SUMMARIES
|
||||||
|
] == sorted((source, folder) for source, folder, _s in EXPECTED_SUMMARIES)
|
||||||
|
|
||||||
|
# --- the pinned tree levels (app.rag.agent's phase-94 templates) -------
|
||||||
|
|
||||||
|
#: ``ls()`` — the top level: sources in registry order (alpha registered
|
||||||
|
#: first), each with its recursive count + stored source-root summary.
|
||||||
|
TOP_HEADER = "2 sources:"
|
||||||
|
TOP_LINES = [
|
||||||
|
f"{ALPHA} — {ALPHA_COUNT} documents",
|
||||||
|
EXPECTED_SUMMARIES[0][2],
|
||||||
|
f"{BETA} — {BETA_COUNT} documents",
|
||||||
|
EXPECTED_SUMMARIES[4][2],
|
||||||
|
]
|
||||||
|
|
||||||
|
#: ``ls(alpha)`` — the source root: the subfolder lines (path order)
|
||||||
|
#: with their stored summaries, then the root's own file line (the
|
||||||
|
#: canonical ``read``/``grep`` identity format, unchanged).
|
||||||
|
SOURCE_HEADER = f"{ALPHA} — 1 documents, 3 folders:"
|
||||||
|
SOURCE_LINES = [
|
||||||
|
f"one/ — 2 documents: {EXPECTED_SUMMARIES[1][2]}",
|
||||||
|
f"two/ — 2 documents: {EXPECTED_SUMMARIES[2][2]}",
|
||||||
|
f"wide/ — {WIDE_COUNT} documents: {EXPECTED_SUMMARIES[3][2]}",
|
||||||
|
f"source: {ALPHA} | path: {ROOT_NOTE} | title: Alpha Root Note",
|
||||||
|
]
|
||||||
|
|
||||||
|
#: ``ls(alpha/two)`` — a leaf folder: the file lines in EXACTLY the
|
||||||
|
#: existing ``source: X | path: Y | title: Z`` format, path order.
|
||||||
|
FOLDER_HEADER = f"{ALPHA}/two — 2 documents, 0 folders:"
|
||||||
|
FOLDER_LINES = [
|
||||||
|
f"source: {ALPHA} | path: {TWO_A} | title: {TWO_A_TITLE}",
|
||||||
|
f"source: {ALPHA} | path: {TWO_B} | title: Alpha Two B",
|
||||||
|
]
|
||||||
|
|
||||||
|
#: ``ls(alpha/wide)`` — the 50-line cap: 51 files → 50 lines (path
|
||||||
|
#: order: wide-01 … wide-50) + one deterministic grep-pointer note;
|
||||||
|
#: wide-51 never reaches the model.
|
||||||
|
WIDE_HEADER = f"{ALPHA}/wide — {WIDE_COUNT} documents, 0 folders:"
|
||||||
|
WIDE_FIRST = f"source: {ALPHA} | path: wide/wide-01.md | title: Wide 01"
|
||||||
|
WIDE_LAST = f"source: {ALPHA} | path: wide/wide-50.md | title: Wide 50"
|
||||||
|
WIDE_NOTE = (
|
||||||
|
"…and 1 more documents in this folder — use grep "
|
||||||
|
"(pattern) to find a specific one."
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- the scripted turns (the mock's ``DRILL_TRIGGER`` questions) -------
|
||||||
|
|
||||||
|
TOP_QUESTION = "Drill down the tree: ls — what sources are indexed?"
|
||||||
|
SOURCE_QUESTION = f"Drill down the tree: ls {ALPHA} — what's in source {ALPHA}?"
|
||||||
|
FOLDER_QUESTION = f"Drill down the tree: ls {ALPHA}/two — list that folder"
|
||||||
|
READ_QUESTION = f"Drill down the tree: read {READ_SP} — read the file"
|
||||||
|
WIDE_QUESTION = f"Drill down the tree: ls {ALPHA}/wide — how many files does this folder hold?"
|
||||||
|
NOPE_QUESTION = f"Drill down the tree: ls {ALPHA}/nope — is there such a folder?"
|
||||||
|
|
||||||
|
#: The read target is a top-2 retrieval document for its question (the
|
||||||
|
#: question names the path — the file self-matches the hybrid gate
|
||||||
|
#: deterministically), so the read gets the phase-72 ALREADY_IN_CONTEXT
|
||||||
|
#: dedupe and the mock answers from the ``<documents>`` prompt with the
|
||||||
|
#: same citation shape, prefixed (the suite pins the dedupe path).
|
||||||
|
READ_ANSWER_PREFIX = f"Already in context: Read {READ_SP}."
|
||||||
|
READ_ANSWER_QUOTE = TWO_A_CONTENT[:80]
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# Fixtures
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def drill_dirs(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, Path]:
|
||||||
|
"""The two-source temp tree (see the module docstring): the app
|
||||||
|
server runs on the same host, so the paths are visible to it. The
|
||||||
|
directory NAMES are the source names (``kind=local`` → the
|
||||||
|
directory's basename, phase 38)."""
|
||||||
|
root = tmp_path_factory.mktemp("bor_ls_tree")
|
||||||
|
alpha = root / ALPHA
|
||||||
|
beta = root / BETA
|
||||||
|
(alpha / "one").mkdir(parents=True)
|
||||||
|
(alpha / "two").mkdir(parents=True)
|
||||||
|
(alpha / "wide").mkdir(parents=True)
|
||||||
|
(beta / "gamma").mkdir(parents=True)
|
||||||
|
|
||||||
|
(alpha / ROOT_NOTE).write_text(
|
||||||
|
_md(
|
||||||
|
"Alpha Root Note",
|
||||||
|
f"{DRILL_LEAD} at the alpha source root: this file sits "
|
||||||
|
"directly under the alpha source, not in any folder.",
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(alpha / "one" / "one-a.md").write_text(
|
||||||
|
_md(
|
||||||
|
"Alpha One A",
|
||||||
|
f"{DRILL_LEAD} for alpha one: this document covers topic A "
|
||||||
|
"of the alpha source tree.",
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(alpha / "one" / "one-b.md").write_text(
|
||||||
|
_md(
|
||||||
|
"Alpha One B",
|
||||||
|
f"{DRILL_LEAD} for alpha one: this document covers topic B "
|
||||||
|
"of the alpha source tree.",
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(alpha / TWO_A).write_text(TWO_A_CONTENT, encoding="utf-8")
|
||||||
|
(alpha / TWO_B).write_text(
|
||||||
|
_md(
|
||||||
|
"Alpha Two B",
|
||||||
|
f"{DRILL_LEAD} for alpha two: this document covers topic B "
|
||||||
|
"of the alpha source tree.",
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
for nn in range(1, WIDE_COUNT + 1):
|
||||||
|
(alpha / "wide" / f"wide-{nn:02d}.md").write_text(
|
||||||
|
_md(
|
||||||
|
f"Wide {nn:02d}",
|
||||||
|
f"One of {WIDE_COUNT} tiny files in the alpha wide "
|
||||||
|
f"folder: {DRILL_LEAD.lower()} line {nn:02d}.",
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(beta / "gamma" / "gamma-a.md").write_text(
|
||||||
|
_md(
|
||||||
|
"Beta Gamma A",
|
||||||
|
f"{DRILL_LEAD} for beta gamma: this document covers topic A "
|
||||||
|
"of the beta source tree.",
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(beta / "gamma" / "gamma-b.md").write_text(
|
||||||
|
_md(
|
||||||
|
"Beta Gamma B",
|
||||||
|
f"{DRILL_LEAD} for beta gamma: this document covers topic B "
|
||||||
|
"of the beta source tree.",
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
assert (alpha / TWO_A).is_file() and (beta / "gamma" / "gamma-b.md").is_file()
|
||||||
|
return alpha, beta
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def app_server(mock_llm: int, drill_dirs: tuple[Path, Path]) -> Iterator[str]:
|
||||||
|
"""The real app under test — per-module app (the conftest pattern,
|
||||||
|
cf. ``test_local_directory_sources.py``): NO ``BOR_GIT_SOURCES``
|
||||||
|
(the env fallback is git-only — the sources here are DB-registered
|
||||||
|
local directories), the mock LLM, the mock-calibrated threshold,
|
||||||
|
and the leak-guarded code defaults. The session app is never
|
||||||
|
started in this isolated run, so no port clash."""
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("DEBUGPY", None)
|
||||||
|
env["BOR_ENVIRONMENT"] = "e2e"
|
||||||
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||||
|
env["BOR_LLM_BASE_URL"] = (
|
||||||
|
"https://aipi.reeseapps.com/v1"
|
||||||
|
if USE_REAL_LLM
|
||||||
|
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||||
|
)
|
||||||
|
# Mock-calibrated threshold (conftest pattern): every scripted
|
||||||
|
# question FTS-matches the fixture docs (the ``drill down the tree``
|
||||||
|
# words), so the gate is HIGH either way.
|
||||||
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||||
|
# Phase 67: instant retry waits + the code-default budget (the
|
||||||
|
# conftest leak-guard pattern).
|
||||||
|
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||||
|
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||||
|
env.setdefault(
|
||||||
|
"BOR_DATABASE_URL",
|
||||||
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||||
|
)
|
||||||
|
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||||
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||||
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||||
|
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||||
|
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||||
|
# beats the .env file): the registry must hold EXACTLY the two
|
||||||
|
# local directories this suite registers (a leftover env git list
|
||||||
|
# would pollute the top-level ``ls`` the whole story asserts on).
|
||||||
|
env["BOR_GIT_SOURCES"] = ""
|
||||||
|
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||||
|
# .env cannot leak corpus-specific settings into the app under test.
|
||||||
|
env["BOR_DOCS_REPO"] = ""
|
||||||
|
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||||
|
_Settings.model_fields["suggestions"].default
|
||||||
|
)
|
||||||
|
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||||
|
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||||
|
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||||
|
cwd=REPO,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_wait_http(f"{APP_URL}/api/health")
|
||||||
|
yield APP_URL
|
||||||
|
finally:
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def app_url(app_server: str) -> str:
|
||||||
|
return app_server
|
||||||
|
|
||||||
|
|
||||||
|
def _truncate_all() -> None:
|
||||||
|
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||||
|
share one Postgres, so a leftover git_sources row would pollute the
|
||||||
|
top-level ``ls`` and a leftover document would show up in the
|
||||||
|
folder listings the drill answers assert on byte-exactly."""
|
||||||
|
with SessionLocal() as db:
|
||||||
|
db.execute(
|
||||||
|
text(
|
||||||
|
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||||
|
"kb_overview, git_sources, folder_summaries"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||||
|
"""Poll the (cookie-authenticated) status endpoint until the run
|
||||||
|
reaches a terminal state (the test_local_directory_sources pattern,
|
||||||
|
over plain httpx — this fixture has no browser page yet)."""
|
||||||
|
deadline = time.monotonic() + timeout_s
|
||||||
|
body: dict[str, Any] = {}
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
r = client.get("/api/sync/status")
|
||||||
|
assert r.status_code == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
if body["state"] in ("success", "failed"):
|
||||||
|
return body
|
||||||
|
time.sleep(0.5)
|
||||||
|
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def synced_kb(app_server: str, drill_dirs: tuple[Path, Path]) -> None:
|
||||||
|
"""The story's precondition: the folder-structured KB synced under
|
||||||
|
the deterministic mock.
|
||||||
|
|
||||||
|
Registers the two temp directories through the authenticated API
|
||||||
|
(the ``test_local_directory_sources.py`` pattern — ``alpha`` FIRST,
|
||||||
|
committed separately, so the registry order — ``(added_at, id)`` —
|
||||||
|
lists alpha before beta, the top-level ``ls`` order the suite
|
||||||
|
asserts), runs the REAL in-process sync (``POST /api/sync`` —
|
||||||
|
walk → chunk → embed → overview → folder summaries → version bump),
|
||||||
|
and pins the stored folder summaries: the mock's canned
|
||||||
|
``FOLDER_SUMMARY_MODE`` branch (task 01) makes the sync store one
|
||||||
|
deterministic row per ≥ 2-doc folder — the drill turns' answers
|
||||||
|
assert on that exact text.
|
||||||
|
"""
|
||||||
|
alpha, beta = drill_dirs
|
||||||
|
_truncate_all()
|
||||||
|
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||||
|
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||||
|
assert r.status_code == 204, r.text
|
||||||
|
r = client.post(
|
||||||
|
"/api/git-sources", json={"kind": "local", "path": str(alpha)}
|
||||||
|
)
|
||||||
|
assert r.status_code == 201, r.text
|
||||||
|
time.sleep(0.05) # distinct added_at: alpha before beta (registry order)
|
||||||
|
r = client.post(
|
||||||
|
"/api/git-sources", json={"kind": "local", "path": str(beta)}
|
||||||
|
)
|
||||||
|
assert r.status_code == 201, r.text
|
||||||
|
r = client.post("/api/sync")
|
||||||
|
assert r.status_code == 202, r.text
|
||||||
|
body = _wait_sync_done_http(client)
|
||||||
|
assert body["state"] == "success", body
|
||||||
|
detail = body["detail"]
|
||||||
|
assert detail["added"] == TOTAL_DOCS, detail
|
||||||
|
assert detail["pruned"] == 0, detail
|
||||||
|
assert detail["overview"] is True, detail
|
||||||
|
# The change-gated folder summaries (phase 94 task 02) landed: one
|
||||||
|
# row per ≥ 2-doc folder, the mock's byte-stable text (the drill
|
||||||
|
# answers quote exactly these lines).
|
||||||
|
with SessionLocal() as db:
|
||||||
|
rows = db.execute(
|
||||||
|
select(
|
||||||
|
FolderSummary.source, FolderSummary.folder_path,
|
||||||
|
FolderSummary.summary,
|
||||||
|
).order_by(FolderSummary.source, FolderSummary.folder_path)
|
||||||
|
).all()
|
||||||
|
assert [(s, f, t) for s, f, t in rows] == EXPECTED_SUMMARIES, rows
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _clean(db_ready: None) -> Iterator[None]:
|
||||||
|
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||||
|
the drill turns never change it, so the folder summaries and the
|
||||||
|
registry persist across the tests of this module)."""
|
||||||
|
with SessionLocal() as db:
|
||||||
|
db.execute(text("TRUNCATE query_log"))
|
||||||
|
db.commit()
|
||||||
|
yield
|
||||||
|
with SessionLocal() as db:
|
||||||
|
db.execute(text("TRUNCATE query_log"))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# Page helpers (the test_agent_document_tools house pattern)
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||||
|
#: (a response clone read in the background) — wire-level assertions
|
||||||
|
#: for the ``tool`` frames, independent of the UI rendering.
|
||||||
|
SSE_HOOK = """
|
||||||
|
() => {
|
||||||
|
if (window.__sseInstalled) return;
|
||||||
|
window.__sseInstalled = true;
|
||||||
|
window.__sseFrames = [];
|
||||||
|
const origFetch = window.fetch;
|
||||||
|
window.fetch = async function (...args) {
|
||||||
|
const res = await origFetch.apply(this, args);
|
||||||
|
try {
|
||||||
|
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||||
|
if (url.includes('/api/chat')) {
|
||||||
|
res.clone().text().then((bodyText) => {
|
||||||
|
for (const block of bodyText.split('\\n\\n')) {
|
||||||
|
const line = block.trim();
|
||||||
|
if (line.startsWith('data: ')) {
|
||||||
|
window.__sseFrames.push(line.slice(6));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
} catch (e) { /* non-clonable responses: ignored */ }
|
||||||
|
return res;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _install_page_hooks(page: Page) -> None:
|
||||||
|
page.evaluate(SSE_HOOK)
|
||||||
|
|
||||||
|
|
||||||
|
def _frames(page: Page) -> list[dict]:
|
||||||
|
"""The SSE frames captured since the last submit (``_submit``
|
||||||
|
clears the buffer), once the hook's background read settles."""
|
||||||
|
deadline = time.monotonic() + 30.0
|
||||||
|
while True:
|
||||||
|
raw = page.evaluate("() => window.__sseFrames || []")
|
||||||
|
parsed = [json.loads(line) for line in raw if line]
|
||||||
|
if any(f.get("type") == "done" for f in parsed):
|
||||||
|
return parsed
|
||||||
|
if time.monotonic() > deadline:
|
||||||
|
raise AssertionError(
|
||||||
|
f"SSE hook captured no `done` frame (frames so far: "
|
||||||
|
f"{len(parsed)}) — hook install failed?"
|
||||||
|
)
|
||||||
|
time.sleep(0.05)
|
||||||
|
|
||||||
|
|
||||||
|
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||||
|
return [f for f in frames if f.get("type") == "tool"]
|
||||||
|
|
||||||
|
|
||||||
|
def _submit(page: Page, question: str) -> None:
|
||||||
|
page.evaluate("window.__sseFrames = []")
|
||||||
|
page.fill("#message-input", question)
|
||||||
|
page.click("#send-btn")
|
||||||
|
# The user bubble lands synchronously with the submit handler.
|
||||||
|
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_settled(page: Page) -> None:
|
||||||
|
"""The turn is complete: answer text in the bubble, button recovered
|
||||||
|
(the phase-48 settle wait, the test_agent_document_tools helper)."""
|
||||||
|
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||||
|
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||||
|
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||||
|
|
||||||
|
|
||||||
|
def _last_brain(page: Page) -> Locator:
|
||||||
|
return page.locator(".msg.brain").last
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_turn(
|
||||||
|
page: Page,
|
||||||
|
expected_tools: list[dict[str, Any]],
|
||||||
|
expected_answer_lines: list[str],
|
||||||
|
) -> None:
|
||||||
|
"""One scripted drill turn, fully asserted: the wire carries exactly
|
||||||
|
the expected ``tool`` frames (ahead of the first ``delta``), the
|
||||||
|
bubble carries the expected answer lines (the mock's echo of the
|
||||||
|
tool results the model received), and the turn was grounded (the
|
||||||
|
``done`` frame is not deflected)."""
|
||||||
|
frames = _frames(page)
|
||||||
|
assert _tool_frames(frames) == expected_tools, _tool_frames(frames)
|
||||||
|
if expected_tools:
|
||||||
|
first_delta = next(
|
||||||
|
i for i, f in enumerate(frames) if f.get("type") == "delta"
|
||||||
|
)
|
||||||
|
assert all(
|
||||||
|
i < first_delta
|
||||||
|
for i, f in enumerate(frames)
|
||||||
|
if f.get("type") == "tool"
|
||||||
|
)
|
||||||
|
done = next(f for f in frames if f.get("type") == "done")
|
||||||
|
assert done["deflected"] is False, done
|
||||||
|
bubble = _last_brain(page).locator(".bubble")
|
||||||
|
for line in expected_answer_lines:
|
||||||
|
expect(bubble).to_contain_text(line)
|
||||||
|
|
||||||
|
|
||||||
|
def _query_log_rows() -> list[QueryLog]:
|
||||||
|
with SessionLocal() as db:
|
||||||
|
return list(db.scalars(select(QueryLog)).all())
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# 1. The scripted drill: sources → folders (summaries) → files → read
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_drill_down_sources_folders_files_and_read(
|
||||||
|
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||||
|
) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
login(page, app_url, next="/")
|
||||||
|
_install_page_hooks(page)
|
||||||
|
|
||||||
|
# --- turn 1: ls() — the top level (sources + source-root summaries) -
|
||||||
|
_submit(page, TOP_QUESTION)
|
||||||
|
_wait_settled(page)
|
||||||
|
lines = _last_brain(page).locator(".tool-call")
|
||||||
|
expect(lines).to_have_count(1)
|
||||||
|
# The no-arg ls line — NOT a scoped "Listing documents in …" one
|
||||||
|
# (regex match: string expectations normalize whitespace, so the
|
||||||
|
# scope check must be a byte-exact pattern).
|
||||||
|
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||||
|
expect(lines.nth(0)).not_to_have_text(re.compile(r"Listing documents in"))
|
||||||
|
_assert_turn(
|
||||||
|
page,
|
||||||
|
[{"type": "tool", "name": "ls", "argument": None}],
|
||||||
|
[TOP_HEADER, *TOP_LINES],
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- turn 2: ls(alpha) — the source root (subfolders + root files) --
|
||||||
|
_submit(page, SOURCE_QUESTION)
|
||||||
|
_wait_settled(page)
|
||||||
|
lines = _last_brain(page).locator(".tool-call")
|
||||||
|
expect(lines).to_have_count(1)
|
||||||
|
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}")
|
||||||
|
_assert_turn(
|
||||||
|
page,
|
||||||
|
[{"type": "tool", "name": "ls", "argument": ALPHA}],
|
||||||
|
[SOURCE_HEADER, *SOURCE_LINES],
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- turn 3: ls(alpha/two) — the folder level (the file lines) ------
|
||||||
|
_submit(page, FOLDER_QUESTION)
|
||||||
|
_wait_settled(page)
|
||||||
|
lines = _last_brain(page).locator(".tool-call")
|
||||||
|
expect(lines).to_have_count(1)
|
||||||
|
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/two")
|
||||||
|
_assert_turn(
|
||||||
|
page,
|
||||||
|
[{"type": "tool", "name": "ls", "argument": f"{ALPHA}/two"}],
|
||||||
|
[FOLDER_HEADER, *FOLDER_LINES],
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- turn 4: read(alpha/two/two-a.md) — the grounded read ------------
|
||||||
|
_submit(page, READ_QUESTION)
|
||||||
|
_wait_settled(page)
|
||||||
|
lines = _last_brain(page).locator(".tool-call")
|
||||||
|
expect(lines).to_have_count(1)
|
||||||
|
# The phase-37 "Reading <source/path>" line (the combined identity).
|
||||||
|
expect(lines.nth(0)).to_contain_text(f"Reading {READ_SP}")
|
||||||
|
frames = _frames(page)
|
||||||
|
assert _tool_frames(frames) == [
|
||||||
|
{"type": "tool", "name": "read", "argument": READ_SP}
|
||||||
|
], _tool_frames(frames)
|
||||||
|
bubble = _last_brain(page).locator(".bubble")
|
||||||
|
# The dedupe notice reached the model (the prefix pins it — the read
|
||||||
|
# was refused as ALREADY_IN_CONTEXT because the target is a top-2
|
||||||
|
# retrieval document)…
|
||||||
|
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
|
||||||
|
# …and the answer still cites the document: the mock quotes the
|
||||||
|
# FIRST 80 chars of the target's text from the ``<documents>``
|
||||||
|
# prompt (the refusal's instruction — answer from that text; the
|
||||||
|
# quote is newline-free, pinned above).
|
||||||
|
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
||||||
|
done = next(f for f in frames if f.get("type") == "done")
|
||||||
|
assert done["deflected"] is False, done
|
||||||
|
# The read document is in the turn's sources (retrieval + agent-read,
|
||||||
|
# deduped — the grounded-turn record).
|
||||||
|
assert any(
|
||||||
|
s["path"] == TWO_A and s["source"] == ALPHA for s in done["sources"]
|
||||||
|
), done["sources"]
|
||||||
|
|
||||||
|
# Durable records: all four turns grounded, in order, the read turn
|
||||||
|
# logging the read document.
|
||||||
|
rows = _query_log_rows()
|
||||||
|
assert [r.question for r in rows] == [
|
||||||
|
TOP_QUESTION, SOURCE_QUESTION, FOLDER_QUESTION, READ_QUESTION
|
||||||
|
]
|
||||||
|
assert all(r.deflected is False for r in rows)
|
||||||
|
assert READ_SP in rows[3].sources, rows[3].sources
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# 2. The 50-line cap: a 51-file folder costs the model 50 lines + the note
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_wide_folder_holds_the_fifty_line_cap(
|
||||||
|
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||||
|
) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
login(page, app_url, next="/")
|
||||||
|
_install_page_hooks(page)
|
||||||
|
|
||||||
|
_submit(page, WIDE_QUESTION)
|
||||||
|
_wait_settled(page)
|
||||||
|
|
||||||
|
lines = _last_brain(page).locator(".tool-call")
|
||||||
|
expect(lines).to_have_count(1)
|
||||||
|
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/wide")
|
||||||
|
|
||||||
|
frames = _frames(page)
|
||||||
|
assert _tool_frames(frames) == [
|
||||||
|
{"type": "tool", "name": "ls", "argument": f"{ALPHA}/wide"}
|
||||||
|
], _tool_frames(frames)
|
||||||
|
done = next(f for f in frames if f.get("type") == "done")
|
||||||
|
assert done["deflected"] is False, done
|
||||||
|
|
||||||
|
# The mock echoed the listing VERBATIM: the header carries the
|
||||||
|
# PRE-cap count (51 — the cap hides lines, not the truth), the
|
||||||
|
# file lines stop at 50, and the one deterministic grep-pointer note
|
||||||
|
# folds the 51st file away.
|
||||||
|
bubble = _last_brain(page).locator(".bubble")
|
||||||
|
text = bubble.text_content() or ""
|
||||||
|
assert WIDE_HEADER in text, text
|
||||||
|
assert WIDE_FIRST in text, text
|
||||||
|
assert WIDE_LAST in text, text
|
||||||
|
assert WIDE_NOTE in text, text
|
||||||
|
assert f"wide/wide-{WIDE_COUNT}.md" not in text, text # the 51st file: gone
|
||||||
|
# Exactly LS_MAX_FILE_LINES file lines reached the model.
|
||||||
|
assert text.count(f"source: {ALPHA} | path: wide/") == LS_MAX_FILE_LINES, text
|
||||||
|
|
||||||
|
row = _query_log_rows()
|
||||||
|
assert len(row) == 1
|
||||||
|
assert row[0].deflected is False
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
# 3. The NOT-A-FOLDER teaching is visible to the model; the scripted
|
||||||
|
# recovery (ls of the parent level) works
|
||||||
|
# --------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_not_a_folder_teaching_and_scripted_recovery(
|
||||||
|
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||||
|
) -> None:
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
login(page, app_url, next="/")
|
||||||
|
_install_page_hooks(page)
|
||||||
|
|
||||||
|
_submit(page, NOPE_QUESTION)
|
||||||
|
_wait_settled(page)
|
||||||
|
|
||||||
|
# Two tool lines: the scripted misuse, then the scripted recovery —
|
||||||
|
# the mock's recovery branch fires ONLY when it RECEIVES the
|
||||||
|
# NOT-A-FOLDER teaching line (``'alpha/nope' is not a folder —
|
||||||
|
# alpha has: one/ two/ wide/`` — the argument echoed, the parent's
|
||||||
|
# subfolders listed): the teaching being visible to the model is
|
||||||
|
# exactly what the second call proves (the phase-72
|
||||||
|
# self-correction contract, now carrying the tree's teaching).
|
||||||
|
lines = _last_brain(page).locator(".tool-call")
|
||||||
|
expect(lines).to_have_count(2)
|
||||||
|
expect(lines.nth(0)).to_contain_text(f"Listing documents in {ALPHA}/nope")
|
||||||
|
expect(lines.nth(1)).to_contain_text(f"Listing documents in {ALPHA}")
|
||||||
|
|
||||||
|
frames = _frames(page)
|
||||||
|
assert _tool_frames(frames) == [
|
||||||
|
{"type": "tool", "name": "ls", "argument": f"{ALPHA}/nope"},
|
||||||
|
{"type": "tool", "name": "ls", "argument": ALPHA},
|
||||||
|
], _tool_frames(frames)
|
||||||
|
done = next(f for f in frames if f.get("type") == "done")
|
||||||
|
assert done["deflected"] is False, done
|
||||||
|
# The loop ended in ONE refusal + ONE correction: the final answer is
|
||||||
|
# the recovery's PARENT listing (not the round-cap, not an echo of
|
||||||
|
# the refusal) — the model self-corrected and got the tree level.
|
||||||
|
bubble = _last_brain(page).locator(".bubble")
|
||||||
|
for line in [SOURCE_HEADER, *SOURCE_LINES]:
|
||||||
|
expect(bubble).to_contain_text(line)
|
||||||
|
|
||||||
|
row = _query_log_rows()
|
||||||
|
assert len(row) == 1
|
||||||
|
assert row[0].deflected is False
|
||||||
@@ -13,11 +13,14 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
|
|||||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||||
deterministic LS-TEACH flow in ``tests/e2e/mock_llm.py``
|
deterministic LS-TEACH flow in ``tests/e2e/mock_llm.py``
|
||||||
(``LS_TEACH_TRIGGER`` — "list the files in this directory" — + the
|
(``LS_TEACH_TRIGGER`` — "list the files in this directory" — + the
|
||||||
HIGH prompt's ``<tools>`` section): the incident's misuse (``ls`` with
|
HIGH prompt's ``<tools>`` section; phase 94: the drill-down ``ls`` —
|
||||||
``{"path": "."}``, id ``call_0``) → the agent's teaching refusal
|
the corrected no-arg listing carries sources only, so the flow drills
|
||||||
(``No source named '.' — check the ls output. (…)``) → the corrected
|
one level before the first file line exists): the incident's misuse
|
||||||
no-arg ``ls()`` (id ``call_1``) → the deterministic
|
(``ls`` with ``{"path": "."}``, id ``call_0``) → the agent's teaching
|
||||||
``These are the indexed documents: <first catalog line>`` answer.
|
refusal (``No source named '.' — check the ls output. (…)``) → the
|
||||||
|
corrected no-arg ``ls()`` (id ``call_1``) → the drill ``ls`` scoped to
|
||||||
|
the first source of the listing (id ``call_2``) → the deterministic
|
||||||
|
``These are the indexed documents: <first file line>`` answer.
|
||||||
|
|
||||||
KB fixture (TRUNCATE-then-seed, house pattern): ONE source with TWO
|
KB fixture (TRUNCATE-then-seed, house pattern): ONE source with TWO
|
||||||
documents of known ``source``/``path``/``title`` (catalog order =
|
documents of known ``source``/``path``/``title`` (catalog order =
|
||||||
@@ -39,21 +42,24 @@ documents of known ``source``/``path``/``title`` (catalog order =
|
|||||||
Test → phase mapping (Playwright Mapping Rule):
|
Test → phase mapping (Playwright Mapping Rule):
|
||||||
1. ``test_ls_misuse_self_corrects_to_noarg_listing`` — the grounded
|
1. ``test_ls_misuse_self_corrects_to_noarg_listing`` — the grounded
|
||||||
LS-TEACH turn: the turn settles (composer re-enables, ``done``
|
LS-TEACH turn: the turn settles (composer re-enables, ``done``
|
||||||
observed), the answer bubble carries the first catalog line — the
|
observed), the answer bubble carries the first file line — the
|
||||||
first document's ``source:`` / ``path:`` / title fields (the
|
first document's ``source:`` / ``path:`` / title fields (the
|
||||||
catalog reached the model and landed in the answer), the UI shows
|
folder listing reached the model and landed in the answer), the UI
|
||||||
the two tool lines (``🔎 Listing documents in <code>.</code>``
|
shows the three tool lines (``🔎 Listing documents in
|
||||||
then ``🔎 Listing documents``), and no error banner. Wire level:
|
<code>.</code>``, ``🔎 Listing documents``, then the drill
|
||||||
the ``tool`` frames arrive in order — first ``ls`` with
|
``🔎 Listing documents in <source>`` — phase 94), and no error
|
||||||
``argument: "."``, then ``ls`` with ``argument: null`` — and there
|
banner. Wire level: the ``tool`` frames arrive in order — first
|
||||||
is NO third ``tool`` frame (the loop ended in one correction, not
|
``ls`` with ``argument: "."``, then ``ls`` with ``argument: null``,
|
||||||
|
then the drill ``ls`` scoped to the source — and there is NO fourth
|
||||||
|
``tool`` frame (the loop ended in one correction + one drill, not
|
||||||
at the round cap).
|
at the round cap).
|
||||||
2. ``test_plain_tool_flow_not_swallowed_by_new_trigger`` — in the SAME
|
2. ``test_plain_tool_flow_not_swallowed_by_new_trigger`` — in the SAME
|
||||||
session, the LS-TEACH turn settles and a follow-up question
|
session, the LS-TEACH turn settles and a follow-up question
|
||||||
carrying ``TOOLS_TRIGGER`` (the single-read flow) still settles
|
carrying ``TOOLS_TRIGGER`` (the single-read flow) still settles
|
||||||
with the read flow's answer (``ls`` → ``read`` on the first
|
with the read flow's answer (``ls`` → the drill ``ls`` (phase 94)
|
||||||
catalog line's combined identity → ``Read <source/path>. <quote>``)
|
→ ``read`` on the first file line's combined identity →
|
||||||
— the new flow did not swallow the existing trigger.
|
``Read <source/path>. <quote>``) — the new flow did not swallow
|
||||||
|
the existing trigger.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -69,7 +75,7 @@ from sqlalchemy import text
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from app.db import SessionLocal
|
from app.db import SessionLocal
|
||||||
from app.models import Chunk, Document
|
from app.models import Chunk, Document, GitSource
|
||||||
from e2e.auth_helpers import login
|
from e2e.auth_helpers import login
|
||||||
from tests.e2e.mock_llm import (
|
from tests.e2e.mock_llm import (
|
||||||
LS_TEACH_TRIGGER,
|
LS_TEACH_TRIGGER,
|
||||||
@@ -211,7 +217,14 @@ def _seed_fixture(db: Session) -> None:
|
|||||||
cosines well past the E2E 0.30 threshold and FTS-matches too →
|
cosines well past the E2E 0.30 threshold and FTS-matches too →
|
||||||
grounded). DOC2 is the seed context only — the single-read flow
|
grounded). DOC2 is the seed context only — the single-read flow
|
||||||
reads the catalog-FIRST document (DOC1), which is not in context.
|
reads the catalog-FIRST document (DOC1), which is not in context.
|
||||||
|
|
||||||
|
Phase 94: the drill-down ``ls`` top level reads the registry —
|
||||||
|
register the source (TRUNCATEd in ``_reset_db_fixture``): the
|
||||||
|
corrected no-arg listing names it, and the drill scopes to it. A
|
||||||
|
non-empty table also ignores the operator's ``BOR_GIT_SOURCES``
|
||||||
|
fallback — deterministic.
|
||||||
"""
|
"""
|
||||||
|
db.add(GitSource(url=SEED_SOURCE, kind="local"))
|
||||||
db.add(
|
db.add(
|
||||||
Document(
|
Document(
|
||||||
source=SEED_SOURCE,
|
source=SEED_SOURCE,
|
||||||
@@ -255,7 +268,10 @@ def _reset_db_fixture() -> None:
|
|||||||
prompts, byte-stable answers."""
|
prompts, byte-stable answers."""
|
||||||
with SessionLocal() as db:
|
with SessionLocal() as db:
|
||||||
db.execute(
|
db.execute(
|
||||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
text(
|
||||||
|
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||||
|
"kb_overview, git_sources"
|
||||||
|
)
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
_seed_fixture(db)
|
_seed_fixture(db)
|
||||||
@@ -382,24 +398,30 @@ def test_ls_misuse_self_corrects_to_noarg_listing(
|
|||||||
expect(bubble).to_contain_text(FIRST_CATALOG_LINE)
|
expect(bubble).to_contain_text(FIRST_CATALOG_LINE)
|
||||||
_assert_no_error_banner(page)
|
_assert_no_error_banner(page)
|
||||||
|
|
||||||
# The UI shows the two tool lines in order: the scoped misuse
|
# The UI shows the three tool lines in order: the scoped misuse
|
||||||
# (🔎 Listing documents in <code>.</code>) then the corrected
|
# (🔎 Listing documents in <code>.</code>), the corrected
|
||||||
# unscoped listing (🔎 Listing documents — no <code>).
|
# unscoped listing (🔎 Listing documents — no <code>), then the
|
||||||
|
# drill (🔎 Listing documents in <source> — phase 94, the top
|
||||||
|
# level lists sources only, so the file lines need one more level).
|
||||||
lines = page.locator(".msg.brain .tool-call")
|
lines = page.locator(".msg.brain .tool-call")
|
||||||
expect(lines).to_have_count(2)
|
expect(lines).to_have_count(3)
|
||||||
expect(lines.nth(0)).to_contain_text("Listing documents in")
|
expect(lines.nth(0)).to_contain_text("Listing documents in")
|
||||||
expect(lines.nth(0).locator("code")).to_have_text(".")
|
expect(lines.nth(0).locator("code")).to_have_text(".")
|
||||||
expect(lines.nth(1)).to_contain_text("Listing documents")
|
expect(lines.nth(1)).to_contain_text("Listing documents")
|
||||||
expect(lines.nth(1).locator("code")).to_have_count(0)
|
expect(lines.nth(1).locator("code")).to_have_count(0)
|
||||||
|
expect(lines.nth(2)).to_contain_text("Listing documents in")
|
||||||
|
expect(lines.nth(2).locator("code")).to_have_text(SEED_SOURCE)
|
||||||
|
|
||||||
# Two rounds on the wire: the tool frames arrive in order — first
|
# Three rounds on the wire: the tool frames arrive in order —
|
||||||
# ls with argument "." (the incident's misuse), then ls with
|
# first ls with argument "." (the incident's misuse), then ls with
|
||||||
# argument null (the correction) — and there is NO third tool
|
# argument null (the correction), then the drill ls scoped to the
|
||||||
# frame: the loop ended in one correction, not at the round cap.
|
# source (phase 94) — and there is NO fourth tool frame: the loop
|
||||||
|
# ended in one correction + one drill, not at the round cap.
|
||||||
frames = _drain_frames(page)
|
frames = _drain_frames(page)
|
||||||
assert _tool_frames(frames) == [
|
assert _tool_frames(frames) == [
|
||||||
{"type": "tool", "name": "ls", "argument": "."},
|
{"type": "tool", "name": "ls", "argument": "."},
|
||||||
{"type": "tool", "name": "ls", "argument": None},
|
{"type": "tool", "name": "ls", "argument": None},
|
||||||
|
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
|
||||||
]
|
]
|
||||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||||
assert all(
|
assert all(
|
||||||
@@ -433,6 +455,7 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
|
|||||||
assert _tool_frames(teach_frames) == [
|
assert _tool_frames(teach_frames) == [
|
||||||
{"type": "tool", "name": "ls", "argument": "."},
|
{"type": "tool", "name": "ls", "argument": "."},
|
||||||
{"type": "tool", "name": "ls", "argument": None},
|
{"type": "tool", "name": "ls", "argument": None},
|
||||||
|
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
|
||||||
]
|
]
|
||||||
expect(
|
expect(
|
||||||
page.locator(".msg.brain .bubble").last
|
page.locator(".msg.brain .bubble").last
|
||||||
@@ -445,14 +468,17 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
|
|||||||
_wait_settled(page)
|
_wait_settled(page)
|
||||||
|
|
||||||
second_msg = page.locator(".msg.brain").last
|
second_msg = page.locator(".msg.brain").last
|
||||||
# The UI shows the single-read flow's two lines: the unscoped ls
|
# The UI shows the single-read flow's three lines: the unscoped ls,
|
||||||
# then the read of the first catalog line's COMBINED identity.
|
# the drill ls (phase 94), then the read of the first file line's
|
||||||
|
# COMBINED identity.
|
||||||
lines = second_msg.locator(".tool-call")
|
lines = second_msg.locator(".tool-call")
|
||||||
expect(lines).to_have_count(2)
|
expect(lines).to_have_count(3)
|
||||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||||
expect(lines.nth(0).locator("code")).to_have_count(0)
|
expect(lines.nth(0).locator("code")).to_have_count(0)
|
||||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||||
expect(lines.nth(1).locator("code")).to_have_text(DOC1_SP)
|
expect(lines.nth(1).locator("code")).to_have_text(SEED_SOURCE)
|
||||||
|
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||||
|
expect(lines.nth(2).locator("code")).to_have_text(DOC1_SP)
|
||||||
|
|
||||||
# The answer quotes the read document (the mock's deterministic
|
# The answer quotes the read document (the mock's deterministic
|
||||||
# echo: "Read <source/path>. <first 80 chars>").
|
# echo: "Read <source/path>. <first 80 chars>").
|
||||||
@@ -461,11 +487,13 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
|
|||||||
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
||||||
_assert_no_error_banner(page)
|
_assert_no_error_banner(page)
|
||||||
|
|
||||||
# Wire level for the follow-up: ls (null) → read (the combined
|
# Wire level for the follow-up: ls (null) → the drill ls (the
|
||||||
# identity) — the single-read flow, unchanged.
|
# source, phase 94) → read (the combined identity) — the
|
||||||
|
# single-read flow, grown by the drill step.
|
||||||
frames = _drain_frames(page)
|
frames = _drain_frames(page)
|
||||||
assert _tool_frames(frames) == [
|
assert _tool_frames(frames) == [
|
||||||
{"type": "tool", "name": "ls", "argument": None},
|
{"type": "tool", "name": "ls", "argument": None},
|
||||||
|
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
|
||||||
{"type": "tool", "name": "read", "argument": DOC1_SP},
|
{"type": "tool", "name": "read", "argument": DOC1_SP},
|
||||||
]
|
]
|
||||||
done = next(f for f in frames if f.get("type") == "done")
|
done = next(f for f in frames if f.get("type") == "done")
|
||||||
|
|||||||
Vendored
+45
-40
File diff suppressed because one or more lines are too long
@@ -1,19 +1,28 @@
|
|||||||
"""Integration: the agent DB accessors against real Postgres (phase 37;
|
"""Integration: the agent DB accessors against real Postgres (phase 37;
|
||||||
the harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
|
the harness-aligned ``ls``/``read``/``grep`` surface, phase 70; the
|
||||||
|
drill-down tree ``ls``, phase 94).
|
||||||
|
|
||||||
``list_catalog`` must order rows by ``(source, path)`` — the same order
|
``_source_document_rows`` must order a source's rows by ``path`` (the
|
||||||
as ``GET /api/docs`` — ``list_source_names`` must resolve the
|
file lines' catalog order), ``list_source_names`` must resolve the
|
||||||
registered source names (the scoped ``ls`` join), and ``find_document``
|
registered source names (the registry join), and ``find_document``
|
||||||
must resolve a hit to the full document row (content included, for the
|
must resolve a hit to the full document row (content included, for the
|
||||||
never-truncated read) and return ``None`` for unknown pairs. Phase 70:
|
never-truncated read) and return ``None`` for unknown pairs. Phase 70:
|
||||||
the ``ls``/``read``/``grep`` tools are pinned here too — the locked
|
the ``ls``/``read``/``grep`` tools are pinned here too — the locked
|
||||||
parameter shape in ``AGENT_TOOLS``, and scripted ``ToolCallPiece``s
|
parameter shape in ``AGENT_TOOLS``, and scripted ``ToolCallPiece``s
|
||||||
executed through ``run_agent`` against the real DB: ``ls`` scoped to a
|
executed through ``run_agent`` against the real DB. Phase 94: the
|
||||||
registered source name (unknown name → refusal), ``read`` on the
|
drill-down ``ls`` against the REAL tables — ``ls()`` lists the
|
||||||
canonical combined ``source/path`` form (first-slash split; a bare
|
registered sources (registry order, recursive counts, stored
|
||||||
source name and an unknown identity get the no-document refusal), and
|
source-root summaries from ``folder_summaries``), ``ls(source)`` /
|
||||||
``grep`` (``all_documents`` for a whole-KB search, ``find_document`` for
|
``ls(source/folder)`` list one folder level (the SQL prefix logic:
|
||||||
a scoped one).
|
subfolders = slash-boundary prefixes, counts = the recursive subtree,
|
||||||
|
file lines in catalog order, capped at 50 + the grep-pointer note),
|
||||||
|
and the refusals (unknown source segment → the no-source refusal; an
|
||||||
|
unknown folder → NOT-A_FOLDER with the parent's subfolders).
|
||||||
|
``read`` runs on the canonical combined ``source/path`` form
|
||||||
|
(first-slash split; a bare source name and an unknown identity get the
|
||||||
|
no-document refusal), and ``grep`` (``all_documents`` for a whole-KB
|
||||||
|
search, ``find_document`` for a scoped one) — both byte-identical
|
||||||
|
across the phase-94 change.
|
||||||
|
|
||||||
Requires: podman compose up -d db
|
Requires: podman compose up -d db
|
||||||
"""
|
"""
|
||||||
@@ -30,7 +39,7 @@ from sqlalchemy import delete, text
|
|||||||
from sqlalchemy.orm import Session
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
from app.config import Settings
|
from app.config import Settings
|
||||||
from app.models import Document, GitSource
|
from app.models import Document, FolderSummary, GitSource
|
||||||
from app.rag import agent
|
from app.rag import agent
|
||||||
from app.rag.agent import AGENT_TOOLS, AgentHolder, run_agent
|
from app.rag.agent import AGENT_TOOLS, AgentHolder, run_agent
|
||||||
from app.rag.llm import LLMClient, RetryPiece, StreamPiece, ToolCallPiece
|
from app.rag.llm import LLMClient, RetryPiece, StreamPiece, ToolCallPiece
|
||||||
@@ -55,11 +64,30 @@ def _doc(db: Session, source: str, path: str, title: str, content: str) -> Docum
|
|||||||
|
|
||||||
@pytest.fixture()
|
@pytest.fixture()
|
||||||
def kb(db) -> Iterator[None]:
|
def kb(db) -> Iterator[None]:
|
||||||
"""Fresh documents table (chunks first — the FK) for these accessors."""
|
"""Fresh documents + folder_summaries tables (chunks first — the FK)
|
||||||
db.execute(text("TRUNCATE chunks, documents"))
|
for these accessors (phase 94: the drill-down ``ls`` reads the
|
||||||
|
stored summaries too)."""
|
||||||
|
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
|
||||||
db.commit()
|
db.commit()
|
||||||
yield
|
yield
|
||||||
db.execute(text("TRUNCATE chunks, documents"))
|
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def registry(db) -> Iterator[None]:
|
||||||
|
"""A FRESH two-source registry (phase 94): the drill-down ``ls``
|
||||||
|
top level IS the registry, so the table is truncated and re-seeded
|
||||||
|
around the tests in a controlled ``(added_at, id)`` order —
|
||||||
|
``Deployments`` before ``Homelab`` (the top-level listing order)."""
|
||||||
|
db.execute(text("TRUNCATE git_sources"))
|
||||||
|
db.commit()
|
||||||
|
db.add(GitSource(url="https://github.com/reese/Deployments.git", kind="git"))
|
||||||
|
db.commit()
|
||||||
|
db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git"))
|
||||||
|
db.commit()
|
||||||
|
yield
|
||||||
|
db.execute(text("TRUNCATE git_sources"))
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
@@ -76,21 +104,24 @@ def src(db) -> Iterator[GitSource]:
|
|||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
def test_list_catalog_orders_by_source_then_path(kb, db) -> None:
|
def test_source_document_rows_order_by_path_within_the_source(kb, db) -> None:
|
||||||
|
"""Phase 94: the file lines' order — the source's rows in ``path``
|
||||||
|
order (the old ``list_catalog``'s per-source ordering, now the
|
||||||
|
``ls`` folder-level accessor's contract; a different source's rows
|
||||||
|
never leak in)."""
|
||||||
_doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
|
_doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
|
||||||
_doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
|
_doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
|
||||||
_doc(db, "Alpha", "c/third.md", "Alpha C", "AC")
|
_doc(db, "Alpha", "c/third.md", "Alpha C", "AC")
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
assert agent.list_catalog(db) == [
|
assert agent._source_document_rows(db, "Zeta") == [
|
||||||
("Alpha", "c/third.md", "Alpha C"),
|
("a/first.md", "Zeta A"),
|
||||||
("Zeta", "a/first.md", "Zeta A"),
|
("b/second.md", "Zeta B"),
|
||||||
("Zeta", "b/second.md", "Zeta B"),
|
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_list_catalog_is_empty_without_rows(kb, db) -> None:
|
def test_source_document_rows_is_empty_without_rows(kb, db) -> None:
|
||||||
assert agent.list_catalog(db) == []
|
assert agent._source_document_rows(db, "Zeta") == []
|
||||||
|
|
||||||
|
|
||||||
def test_list_source_names_resolves_registry_rows(db) -> None:
|
def test_list_source_names_resolves_registry_rows(db) -> None:
|
||||||
@@ -238,30 +269,156 @@ async def _consume(
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
# ---------- ls (scoped through the real registry) ----------
|
# ---------- ls (the drill-down tree, phase 94 — the real registry + DB) ----------
|
||||||
|
|
||||||
|
|
||||||
def test_ls_scoped_to_registered_source_through_run_agent(kb, src, db) -> None:
|
def test_ls_top_level_lists_registered_sources_through_run_agent(
|
||||||
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
|
kb, registry, db
|
||||||
_doc(db, "Other", "b.md", "B", "B-CONTENT")
|
) -> None:
|
||||||
|
"""No path: the TOP level against the real tables — registry order
|
||||||
|
(``(added_at, id)`` — Deployments before Homelab), recursive counts
|
||||||
|
(all of a source's documents), the stored source-root summary
|
||||||
|
(``folder_path = ''``) shown only when stored."""
|
||||||
|
_doc(db, "Deployments", "a/one.md", "A1", "A1-CONTENT")
|
||||||
|
_doc(db, "Homelab", "x.md", "X", "X-CONTENT")
|
||||||
|
_doc(db, "Homelab", "y/z.md", "Z", "Z-CONTENT")
|
||||||
|
db.add(FolderSummary(source="Homelab", folder_path="", summary="Home lab notes."))
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
holder, llm = _run_call(db, "ls", {"path": "Homelab"})
|
holder, llm = _run_call(db, "ls", {})
|
||||||
|
|
||||||
# Offered: the first request carries AGENT_TOOLS (the 3-tool list).
|
|
||||||
assert llm.requests[0][1] == AGENT_TOOLS
|
assert llm.requests[0][1] == AGENT_TOOLS
|
||||||
# Executed against the real DB: the listing filtered to the source.
|
|
||||||
assert llm.requests[1][0][3]["content"] == (
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
"1 documents:\nsource: Homelab | path: a.md | title: A"
|
"2 sources:\n"
|
||||||
|
"\n"
|
||||||
|
"Deployments — 1 documents\n"
|
||||||
|
"Homelab — 2 documents\n"
|
||||||
|
" Home lab notes."
|
||||||
)
|
)
|
||||||
assert holder.tool_calls == 1
|
assert holder.tool_calls == 1
|
||||||
assert holder.read_docs == []
|
assert holder.read_docs == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_top_level_empty_registry_through_run_agent(
|
||||||
|
kb, db, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""No registered sources: the top level is the header line alone
|
||||||
|
(``0 sources:`` — the old ``0 documents:`` behavior preserved in
|
||||||
|
spirit), still counted. The env fallback (``BOR_GIT_SOURCES`` — the
|
||||||
|
operator's ``.env`` may name sources) is emptied for the test, so
|
||||||
|
the registry is genuinely empty."""
|
||||||
|
import app.rag.git_sources as git_sources_mod
|
||||||
|
|
||||||
|
db.execute(text("TRUNCATE git_sources"))
|
||||||
|
db.commit()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
git_sources_mod,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings(git_sources=""),
|
||||||
|
)
|
||||||
|
_doc(db, "Orphan", "a.md", "A", "A-CONTENT") # indexed but unregistered
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
holder, llm = _run_call(db, "ls", {})
|
||||||
|
assert llm.requests[1][0][3]["content"] == "0 sources:"
|
||||||
|
assert holder.tool_calls == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_source_scope_lists_root_folder_through_run_agent(kb, registry, db) -> None:
|
||||||
|
"""A registered source name: the source's ROOT folder — the direct
|
||||||
|
subfolders (path order, recursive counts, stored summaries attached)
|
||||||
|
+ the root's own file lines in catalog order — against the real
|
||||||
|
tables; a registered source with no documents lists its header
|
||||||
|
line alone."""
|
||||||
|
_doc(db, "Homelab", "backups/cron.md", "Cron", "CRON")
|
||||||
|
_doc(db, "Homelab", "backups/restic.md", "Restic", "RESTIC")
|
||||||
|
_doc(db, "Homelab", "networking/lan.md", "LAN", "LAN")
|
||||||
|
_doc(db, "Homelab", "readme.md", "Readme", "README")
|
||||||
|
db.add(
|
||||||
|
FolderSummary(
|
||||||
|
source="Homelab", folder_path="backups", summary="Backup notes."
|
||||||
|
)
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
holder, llm = _run_call(db, "ls", {"path": "Homelab"})
|
||||||
|
|
||||||
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
|
"Homelab — 1 documents, 2 folders:\n"
|
||||||
|
"\n"
|
||||||
|
" backups/ — 2 documents: Backup notes.\n"
|
||||||
|
" networking/ — 1 documents\n"
|
||||||
|
"\n"
|
||||||
|
"source: Homelab | path: readme.md | title: Readme"
|
||||||
|
)
|
||||||
|
assert holder.tool_calls == 1
|
||||||
|
assert holder.read_docs == []
|
||||||
|
|
||||||
|
# A registered source with no documents: the header line alone.
|
||||||
|
holder0, llm0 = _run_call(db, "ls", {"path": "Deployments"})
|
||||||
|
assert llm0.requests[1][0][3]["content"] == "Deployments — 0 documents, 0 folders:"
|
||||||
|
assert holder0.tool_calls == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_nested_folder_scope_drills_one_level_through_run_agent(
|
||||||
|
kb, registry, db
|
||||||
|
) -> None:
|
||||||
|
"""A ``source/folder`` path: that folder's subfolders + own file
|
||||||
|
lines (identity = ``source/folder``) — the drill-down against the
|
||||||
|
real tables."""
|
||||||
|
_doc(db, "Homelab", "networking/lan/a.md", "A", "A")
|
||||||
|
_doc(db, "Homelab", "networking/lan/b.md", "B", "B")
|
||||||
|
_doc(db, "Homelab", "networking/vpn/c.md", "C", "C")
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
holder, llm = _run_call(db, "ls", {"path": "Homelab/networking"})
|
||||||
|
|
||||||
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
|
"Homelab/networking — 0 documents, 2 folders:\n"
|
||||||
|
"\n"
|
||||||
|
" networking/lan/ — 2 documents\n"
|
||||||
|
" networking/vpn/ — 1 documents"
|
||||||
|
)
|
||||||
|
assert holder.tool_calls == 1
|
||||||
|
assert holder.read_docs == []
|
||||||
|
|
||||||
|
# One level deeper.
|
||||||
|
holder2, llm2 = _run_call(db, "ls", {"path": "Homelab/networking/lan"})
|
||||||
|
assert llm2.requests[1][0][3]["content"] == (
|
||||||
|
"Homelab/networking/lan — 2 documents, 0 folders:\n"
|
||||||
|
"\n"
|
||||||
|
"source: Homelab | path: networking/lan/a.md | title: A\n"
|
||||||
|
"source: Homelab | path: networking/lan/b.md | title: B"
|
||||||
|
)
|
||||||
|
assert holder2.tool_calls == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_folder_file_cap_through_run_agent(kb, registry, db) -> None:
|
||||||
|
"""The cap end-to-end: 51 direct files in one folder cost 50 file
|
||||||
|
lines + the deterministic grep-pointer note, never 51."""
|
||||||
|
for i in range(51):
|
||||||
|
_doc(db, "Homelab", f"big/f{i:03d}.md", f"T{i}", "BODY")
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
holder, llm = _run_call(db, "ls", {"path": "Homelab/big"})
|
||||||
|
|
||||||
|
content = llm.requests[1][0][3]["content"]
|
||||||
|
lines = content.splitlines()
|
||||||
|
assert lines[0] == "Homelab/big — 51 documents, 0 folders:"
|
||||||
|
assert lines[2] == "source: Homelab | path: big/f000.md | title: T0"
|
||||||
|
assert lines[51] == "source: Homelab | path: big/f049.md | title: T49"
|
||||||
|
assert lines[52] == (
|
||||||
|
"…and 1 more documents in this folder — use grep (pattern) to "
|
||||||
|
"find a specific one."
|
||||||
|
)
|
||||||
|
assert len(lines) == 53
|
||||||
|
assert holder.tool_calls == 1
|
||||||
|
|
||||||
|
|
||||||
def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None:
|
def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None:
|
||||||
"""Phase 72: the no-source refusal now carries the teaching
|
"""A ``path`` without ``/`` matching no source name is a refusal —
|
||||||
parenthetical — the prefix byte-identical to the pre-phase-72 line;
|
the extended line with the teaching parenthetical (phase 72, the
|
||||||
still not counted."""
|
prefix byte-identical to the pre-phase-72 line); still not counted."""
|
||||||
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
|
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
@@ -274,11 +431,11 @@ def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None
|
|||||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||||
|
|
||||||
|
|
||||||
def test_ls_path_like_scope_teaching_refusal_through_run_agent(kb, src, db) -> None:
|
def test_ls_path_like_scope_unknown_source_gets_no_source_refusal(kb, src, db) -> None:
|
||||||
"""Phase 72: a ``/``-containing ``path`` is a document path, not a
|
"""Phase 94: a ``/`` now names a folder — the phase-72 document-path
|
||||||
source name — the ``LS_PATH_NOT_A_SOURCE`` teaching line (no
|
teaching is DELETED; a ``source/…`` argument whose FIRST segment
|
||||||
registry lookup needed), not counted, the tools stay offered on the
|
names no registered source gets the no-source refusal (the segment
|
||||||
next request."""
|
echoed), not counted, the tools stay offered."""
|
||||||
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
|
_doc(db, "Homelab", "a.md", "A", "A-CONTENT")
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
@@ -286,7 +443,30 @@ def test_ls_path_like_scope_teaching_refusal_through_run_agent(kb, src, db) -> N
|
|||||||
|
|
||||||
assert (
|
assert (
|
||||||
llm.requests[1][0][3]["content"]
|
llm.requests[1][0][3]["content"]
|
||||||
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
|
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
|
||||||
|
)
|
||||||
|
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||||
|
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_unknown_folder_gets_not_a_folder_with_parents_subfolders(
|
||||||
|
kb, src, db,
|
||||||
|
) -> None:
|
||||||
|
"""Phase 94: a folder segment matching no indexed prefix gets the
|
||||||
|
NOT-A_FOLDER teaching — the argument echoed, the source named, its
|
||||||
|
DIRECT subfolders listed (the self-correction list), not counted,
|
||||||
|
the tools stay offered."""
|
||||||
|
_doc(db, "Homelab", "backups/cron.md", "Cron", "CRON")
|
||||||
|
_doc(db, "Homelab", "containers/caddy.md", "Caddy", "CADDY")
|
||||||
|
_doc(db, "Homelab", "networking/lan.md", "LAN", "LAN")
|
||||||
|
_doc(db, "Homelab", "readme.md", "Readme", "README")
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
holder, llm = _run_call(db, "ls", {"path": "Homelab/netwoking"})
|
||||||
|
|
||||||
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
|
"'Homelab/netwoking' is not a folder — Homelab has: "
|
||||||
|
"backups/ containers/ networking/"
|
||||||
)
|
)
|
||||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||||
|
|||||||
@@ -943,7 +943,10 @@ def test_tool_execution_db_failure_yields_error_event(
|
|||||||
def boom(*_a: Any, **_k: Any) -> Any:
|
def boom(*_a: Any, **_k: Any) -> Any:
|
||||||
raise RuntimeError("db exploded mid tool call")
|
raise RuntimeError("db exploded mid tool call")
|
||||||
|
|
||||||
monkeypatch.setattr(agent, "list_catalog", boom)
|
# Phase 94: the no-arg ``ls`` executes through ``ls_top`` — the
|
||||||
|
# failure hook moves with the rewrite (same contract: the tool
|
||||||
|
# frame goes out first, the structured error ends the turn).
|
||||||
|
monkeypatch.setattr(agent, "ls_top", boom)
|
||||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
|
||||||
try:
|
try:
|
||||||
_, _, frames = _stream_chat(client, QUESTION)
|
_, _, frames = _stream_chat(client, QUESTION)
|
||||||
|
|||||||
@@ -104,6 +104,21 @@ def _stub_bump(monkeypatch: pytest.MonkeyPatch) -> list[None]:
|
|||||||
return bumps
|
return bumps
|
||||||
|
|
||||||
|
|
||||||
|
def _stub_folder_summaries(monkeypatch: pytest.MonkeyPatch) -> list[dict]:
|
||||||
|
"""Stub the phase-94 folder-summary regeneration (this file keeps
|
||||||
|
its no-real-DB / no-network style — the real generator would read
|
||||||
|
the global ``documents`` table and burn ``lite`` calls). Returns
|
||||||
|
the call record; the canned stats are the zero dict."""
|
||||||
|
calls: list[dict] = []
|
||||||
|
|
||||||
|
async def fake_generate(db: object, llm: object, *, skip: bool = False) -> dict[str, int]:
|
||||||
|
calls.append({"skip": skip})
|
||||||
|
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||||
|
|
||||||
|
monkeypatch.setattr(import_docs, "generate_folder_summaries", fake_generate)
|
||||||
|
return calls
|
||||||
|
|
||||||
|
|
||||||
# --- repo_name -------------------------------------------------------------
|
# --- repo_name -------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
@@ -293,6 +308,7 @@ def test_main_rows_branch_passes_ignore_map_to_import(
|
|||||||
fake_import = FakeImportSources()
|
fake_import = FakeImportSources()
|
||||||
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
||||||
_stub_bump(monkeypatch)
|
_stub_bump(monkeypatch)
|
||||||
|
_stub_folder_summaries(monkeypatch)
|
||||||
|
|
||||||
rc = import_docs.main([])
|
rc = import_docs.main([])
|
||||||
|
|
||||||
@@ -329,6 +345,7 @@ def test_main_git_sources_clone_then_import(
|
|||||||
fake_import = FakeImportSources()
|
fake_import = FakeImportSources()
|
||||||
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
||||||
bumps = _stub_bump(monkeypatch)
|
bumps = _stub_bump(monkeypatch)
|
||||||
|
_stub_folder_summaries(monkeypatch)
|
||||||
|
|
||||||
rc = import_docs.main([])
|
rc = import_docs.main([])
|
||||||
|
|
||||||
@@ -371,6 +388,7 @@ def test_main_cli_source_still_imports_manual_dir(
|
|||||||
fake_import = FakeImportSources()
|
fake_import = FakeImportSources()
|
||||||
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
||||||
bumps = _stub_bump(monkeypatch)
|
bumps = _stub_bump(monkeypatch)
|
||||||
|
_stub_folder_summaries(monkeypatch)
|
||||||
|
|
||||||
rc = import_docs.main(["--source", str(manual)])
|
rc = import_docs.main(["--source", str(manual)])
|
||||||
|
|
||||||
|
|||||||
@@ -23,6 +23,15 @@ and unchanged re-runs never bump (``sources_version=skipped``), and a
|
|||||||
failed ``lite`` never rolls the bump back. The counter is pinned to the
|
failed ``lite`` never rolls the bump back. The counter is pinned to the
|
||||||
migration-0010 seed (0) around every test by
|
migration-0010 seed (0) around every test by
|
||||||
:func:`_reset_sources_version`.
|
:func:`_reset_sources_version`.
|
||||||
|
|
||||||
|
Phase 94 (task 02, line-extension house rule): the summary line now
|
||||||
|
ends with the folder-summary stats —
|
||||||
|
``folder_summaries=<generated>/<failed>/<pruned>`` when the gate fired
|
||||||
|
(this fixture's 2-doc source holds exactly ONE qualifying subtree: the
|
||||||
|
source root) or ``folder_summaries=skipped`` otherwise — so the line
|
||||||
|
pinned here gains that token, and a KB-changing run burns exactly ONE
|
||||||
|
extra ``lite`` call (the source-root folder summary, markdown files
|
||||||
|
never get a document summary).
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -98,10 +107,10 @@ def src(tmp_path: Path) -> Path:
|
|||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
@pytest.fixture(autouse=True)
|
||||||
def _clean_kb(db: Session) -> Iterator[None]:
|
def _clean_kb(db: Session) -> Iterator[None]:
|
||||||
db.execute(text("TRUNCATE chunks, documents, kb_overview"))
|
db.execute(text("TRUNCATE chunks, documents, kb_overview, folder_summaries"))
|
||||||
db.commit()
|
db.commit()
|
||||||
yield
|
yield
|
||||||
db.execute(text("TRUNCATE chunks, documents, kb_overview"))
|
db.execute(text("TRUNCATE chunks, documents, kb_overview, folder_summaries"))
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
@@ -149,16 +158,24 @@ def test_changed_import_writes_overview_row(
|
|||||||
|
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert "added=2" in out
|
assert "added=2" in out
|
||||||
assert out.rstrip().endswith("overview=updated sources_version=1")
|
# Phase 94: the line gains the folder-stats token — the 2-doc source
|
||||||
|
# holds one qualifying subtree (the source root): 1 generated.
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=updated sources_version=1 folder_summaries=1/0/0"
|
||||||
|
)
|
||||||
assert _version(db) == 1 # phase 53: a changed import bumps exactly once
|
assert _version(db) == 1 # phase 53: a changed import bumps exactly once
|
||||||
# Exactly one lite call — the overview itself (markdown files never
|
# Exactly two lite calls — the overview + the source-root folder
|
||||||
# get a summary, so nothing else may touch ``chat``).
|
# summary (markdown files never get a document summary, so nothing
|
||||||
assert len(llm.chat_calls) == 1
|
# else may touch ``chat``).
|
||||||
|
assert len(llm.chat_calls) == 2
|
||||||
by_role = {m["role"]: m["content"] for m in llm.chat_calls[0]}
|
by_role = {m["role"]: m["content"] for m in llm.chat_calls[0]}
|
||||||
assert "KB_OVERVIEW_MODE" in by_role["system"]
|
assert "KB_OVERVIEW_MODE" in by_role["system"]
|
||||||
# One line per doc: source — path — title (no summary for markdown).
|
# One line per doc: source — path — title (no summary for markdown).
|
||||||
assert "MyDocs — alpha.md — Alpha" in by_role["user"]
|
assert "MyDocs — alpha.md — Alpha" in by_role["user"]
|
||||||
assert "MyDocs — beta.md — Beta" in by_role["user"]
|
assert "MyDocs — beta.md — Beta" in by_role["user"]
|
||||||
|
by_role = {m["role"]: m["content"] for m in llm.chat_calls[1]}
|
||||||
|
assert "FOLDER_SUMMARY_MODE" in by_role["system"]
|
||||||
|
assert by_role["user"].splitlines()[0] == "Folder: MyDocs"
|
||||||
# The model's outline lands in the single row.
|
# The model's outline lands in the single row.
|
||||||
row = _row(db)
|
row = _row(db)
|
||||||
assert row is not None
|
assert row is not None
|
||||||
@@ -177,16 +194,20 @@ def test_unchanged_reimport_does_not_call_lite(
|
|||||||
llm = FakeEmbedder()
|
llm = FakeEmbedder()
|
||||||
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert out.rstrip().endswith("overview=updated sources_version=1")
|
assert out.rstrip().endswith(
|
||||||
assert len(llm.chat_calls) == 1
|
"overview=updated sources_version=1 folder_summaries=1/0/0"
|
||||||
|
)
|
||||||
|
assert len(llm.chat_calls) == 2 # overview + source-root folder summary
|
||||||
assert _row(db) is not None
|
assert _row(db) is not None
|
||||||
|
|
||||||
# Same hashes → no KB change → no lite call, previous outline kept.
|
# Same hashes → no KB change → no lite call, previous outline kept.
|
||||||
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert "unchanged=2" in out
|
assert "unchanged=2" in out
|
||||||
assert out.rstrip().endswith("overview=skipped sources_version=skipped")
|
assert out.rstrip().endswith(
|
||||||
assert len(llm.chat_calls) == 1 # no new lite call
|
"overview=skipped sources_version=skipped folder_summaries=skipped"
|
||||||
|
)
|
||||||
|
assert len(llm.chat_calls) == 2 # no new lite call
|
||||||
row = _row(db)
|
row = _row(db)
|
||||||
assert row is not None and row.content == "Summary of MyDocs"
|
assert row is not None and row.content == "Summary of MyDocs"
|
||||||
assert _version(db) == 1 # phase 53: an unchanged re-run never bumps
|
assert _version(db) == 1 # phase 53: an unchanged re-run never bumps
|
||||||
@@ -201,7 +222,9 @@ def test_lite_failure_is_fail_soft(
|
|||||||
good = FakeEmbedder()
|
good = FakeEmbedder()
|
||||||
rc, out = _run_main(monkeypatch, good, ["--source", str(src)], capsys)
|
rc, out = _run_main(monkeypatch, good, ["--source", str(src)], capsys)
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert out.rstrip().endswith("overview=updated sources_version=1")
|
assert out.rstrip().endswith(
|
||||||
|
"overview=updated sources_version=1 folder_summaries=1/0/0"
|
||||||
|
)
|
||||||
previous = _row(db)
|
previous = _row(db)
|
||||||
assert previous is not None
|
assert previous is not None
|
||||||
previous_content = previous.content
|
previous_content = previous.content
|
||||||
@@ -213,8 +236,12 @@ def test_lite_failure_is_fail_soft(
|
|||||||
rc, out = _run_main(monkeypatch, bad, ["--source", str(src)], capsys)
|
rc, out = _run_main(monkeypatch, bad, ["--source", str(src)], capsys)
|
||||||
assert rc == 0 # a failed outline must not fail the import
|
assert rc == 0 # a failed outline must not fail the import
|
||||||
assert "updated=1" in out
|
assert "updated=1" in out
|
||||||
assert out.rstrip().endswith("overview=failed sources_version=2")
|
# Phase 94: the folder batch fails too (per-folder fail-soft) — the
|
||||||
assert len(bad.chat_calls) == 1 # the (failed) attempt was made
|
# failed attempt counts into the stats, the previous row stays.
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=failed sources_version=2 folder_summaries=0/1/0"
|
||||||
|
)
|
||||||
|
assert len(bad.chat_calls) == 2 # the (failed) attempts were made
|
||||||
row = _row(db)
|
row = _row(db)
|
||||||
assert row is not None
|
assert row is not None
|
||||||
assert row.content == previous_content # previous row untouched
|
assert row.content == previous_content # previous row untouched
|
||||||
@@ -232,8 +259,10 @@ def test_limit_run_skips_overview(
|
|||||||
llm = FakeEmbedder()
|
llm = FakeEmbedder()
|
||||||
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert out.rstrip().endswith("overview=updated sources_version=1")
|
assert out.rstrip().endswith(
|
||||||
assert len(llm.chat_calls) == 1
|
"overview=updated sources_version=1 folder_summaries=1/0/0"
|
||||||
|
)
|
||||||
|
assert len(llm.chat_calls) == 2
|
||||||
|
|
||||||
# An incomplete walk must not rewrite the outline (mirrors the
|
# An incomplete walk must not rewrite the outline (mirrors the
|
||||||
# --prune-with---limit guard) — and must not advance the version.
|
# --prune-with---limit guard) — and must not advance the version.
|
||||||
@@ -241,8 +270,10 @@ def test_limit_run_skips_overview(
|
|||||||
rc, out = _run_main(monkeypatch, llm, ["--source", str(src), "--limit", "1"], capsys)
|
rc, out = _run_main(monkeypatch, llm, ["--source", str(src), "--limit", "1"], capsys)
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert "updated=1" in out
|
assert "updated=1" in out
|
||||||
assert out.rstrip().endswith("overview=skipped sources_version=skipped")
|
assert out.rstrip().endswith(
|
||||||
assert len(llm.chat_calls) == 1 # --limit never burns a lite call
|
"overview=skipped sources_version=skipped folder_summaries=skipped"
|
||||||
|
)
|
||||||
|
assert len(llm.chat_calls) == 2 # --limit never burns a lite call
|
||||||
row = _row(db)
|
row = _row(db)
|
||||||
assert row is not None and row.content == "Summary of MyDocs"
|
assert row is not None and row.content == "Summary of MyDocs"
|
||||||
assert _version(db) == 1 # phase 53: --limit debug runs never bump
|
assert _version(db) == 1 # phase 53: --limit debug runs never bump
|
||||||
@@ -262,7 +293,9 @@ def test_empty_source_without_row_creates_nothing(
|
|||||||
|
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert "files=0" in out
|
assert "files=0" in out
|
||||||
assert out.rstrip().endswith("overview=skipped sources_version=skipped")
|
assert out.rstrip().endswith(
|
||||||
|
"overview=skipped sources_version=skipped folder_summaries=skipped"
|
||||||
|
)
|
||||||
assert llm.chat_calls == [] # no KB → no outline, no wasted model call
|
assert llm.chat_calls == [] # no KB → no outline, no wasted model call
|
||||||
assert _row(db) is None # nothing created
|
assert _row(db) is None # nothing created
|
||||||
assert _version(db) == 0 # nothing changed → nothing bumped
|
assert _version(db) == 0 # nothing changed → nothing bumped
|
||||||
@@ -283,7 +316,9 @@ def test_prune_only_run_bumps_sources_version(
|
|||||||
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert "added=2" in out
|
assert "added=2" in out
|
||||||
assert out.rstrip().endswith("overview=updated sources_version=1")
|
assert out.rstrip().endswith(
|
||||||
|
"overview=updated sources_version=1 folder_summaries=1/0/0"
|
||||||
|
)
|
||||||
assert _version(db) == 1
|
assert _version(db) == 1
|
||||||
|
|
||||||
# Delete one file; a --prune run drops exactly it: no add/update,
|
# Delete one file; a --prune run drops exactly it: no add/update,
|
||||||
@@ -292,6 +327,12 @@ def test_prune_only_run_bumps_sources_version(
|
|||||||
rc, out = _run_main(monkeypatch, llm, ["--source", str(src), "--prune"], capsys)
|
rc, out = _run_main(monkeypatch, llm, ["--source", str(src), "--prune"], capsys)
|
||||||
assert rc == 0
|
assert rc == 0
|
||||||
assert "pruned=1" in out
|
assert "pruned=1" in out
|
||||||
assert out.rstrip().endswith("overview=skipped sources_version=2")
|
# Phase 94: the folder gate is the overview's (added + updated > 0
|
||||||
|
# or empty table) — a prune-only re-walk with a populated table
|
||||||
|
# skips generation (the remaining 1-doc source stays summarized by
|
||||||
|
# its existing root row, which still describes it).
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=skipped sources_version=2 folder_summaries=skipped"
|
||||||
|
)
|
||||||
assert _version(db) == 2 # the prune-only change bumped exactly once
|
assert _version(db) == 2 # the prune-only change bumped exactly once
|
||||||
assert _row(db) is not None # the outline row is untouched
|
assert _row(db) is not None # the outline row is untouched
|
||||||
|
|||||||
@@ -0,0 +1,248 @@
|
|||||||
|
"""Integration: migration 0017 (folder_summaries) schema contract
|
||||||
|
(phase 94, task 01).
|
||||||
|
|
||||||
|
Drives the **real Alembic engine** against the live dev database
|
||||||
|
(``podman compose up -d db``), mirroring the house pattern of
|
||||||
|
``test_migration_0016.py`` (information_schema assertions on the state
|
||||||
|
the migration must leave). The tests target the 0016 → 0017 step
|
||||||
|
explicitly so later migrations cannot break them:
|
||||||
|
|
||||||
|
* upgrade 0016 → 0017 → the ``folder_summaries`` table exists with the
|
||||||
|
full column contract — PK ``(source, folder_path)`` (VARCHAR(120) /
|
||||||
|
VARCHAR(1000) NOT NULL, mirroring ``documents.source`` /
|
||||||
|
``documents.path``), ``summary`` TEXT NOT NULL, ``updated_at``
|
||||||
|
TIMESTAMPTZ NOT NULL with the now() server default (house style) —
|
||||||
|
while the 0016 ``ui_settings`` schema survives;
|
||||||
|
* inserted rows round-trip their values (a source-root row with
|
||||||
|
``folder_path = ''`` and a nested-folder row);
|
||||||
|
* downgrade to 0016 → the table is GONE (A13 — reversible), the rest of
|
||||||
|
the schema (``ui_settings`` + ``api_tokens``) survives;
|
||||||
|
* upgrade back to 0017 → the table is back (round-trip).
|
||||||
|
|
||||||
|
The ``alembic`` fixture guarantees the DB ends at head even if a test
|
||||||
|
fails or the process is interrupted.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from alembic.config import Config
|
||||||
|
from sqlalchemy import text
|
||||||
|
from sqlalchemy.exc import IntegrityError
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from alembic import command
|
||||||
|
from app.db import db_available
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def alembic(db: Session) -> Iterator[Config]:
|
||||||
|
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||||
|
|
||||||
|
Starts at head (repairs an interrupted earlier run); teardown upgrades
|
||||||
|
to head no matter what happened, so the dev DB is never left below
|
||||||
|
head.
|
||||||
|
"""
|
||||||
|
if not db_available():
|
||||||
|
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||||
|
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||||
|
cfg.set_main_option("script_location", "alembic")
|
||||||
|
command.upgrade(cfg, "head")
|
||||||
|
try:
|
||||||
|
yield cfg
|
||||||
|
finally:
|
||||||
|
command.upgrade(cfg, "head")
|
||||||
|
|
||||||
|
|
||||||
|
def _version(db: Session) -> str | None:
|
||||||
|
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||||
|
|
||||||
|
|
||||||
|
def _table_exists(db: Session, table: str) -> bool:
|
||||||
|
return (
|
||||||
|
db.execute(
|
||||||
|
text("SELECT 1 FROM information_schema.tables WHERE table_name = :t"),
|
||||||
|
{"t": table},
|
||||||
|
).scalar()
|
||||||
|
is not None
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _column(db: Session, table: str, column: str) -> tuple[Any, ...] | None:
|
||||||
|
"""(data_type, is_nullable, column_default, character_maximum_length)
|
||||||
|
for one table column."""
|
||||||
|
row = db.execute(
|
||||||
|
text(
|
||||||
|
"SELECT data_type, is_nullable, column_default, character_maximum_length"
|
||||||
|
" FROM information_schema.columns"
|
||||||
|
" WHERE table_name = :t AND column_name = :c"
|
||||||
|
),
|
||||||
|
{"t": table, "c": column},
|
||||||
|
).fetchone()
|
||||||
|
return tuple(row) if row is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
def _pk_columns(db: Session, table: str) -> list[str]:
|
||||||
|
"""The table's PRIMARY KEY columns in ordinal position."""
|
||||||
|
rows = db.execute(
|
||||||
|
text(
|
||||||
|
"SELECT kcu.column_name"
|
||||||
|
" FROM information_schema.table_constraints tc"
|
||||||
|
" JOIN information_schema.key_column_usage kcu"
|
||||||
|
" ON kcu.constraint_name = tc.constraint_name"
|
||||||
|
" AND kcu.table_name = tc.table_name"
|
||||||
|
" WHERE tc.table_name = :t"
|
||||||
|
" AND tc.constraint_type = 'PRIMARY KEY'"
|
||||||
|
" ORDER BY kcu.ordinal_position"
|
||||||
|
),
|
||||||
|
{"t": table},
|
||||||
|
).fetchall()
|
||||||
|
return [r[0] for r in rows]
|
||||||
|
|
||||||
|
|
||||||
|
def _clear_rows(db: Session) -> None:
|
||||||
|
db.execute(text("DELETE FROM folder_summaries"))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_to_0017_creates_folder_summaries(db: Session, alembic: Config) -> None:
|
||||||
|
"""Upgrade 0016 → 0017: the table exists with the full column
|
||||||
|
contract (PK ``(source, folder_path)`` mirroring
|
||||||
|
``documents.source`` / ``documents.path``; TEXT summary NOT NULL;
|
||||||
|
TIMESTAMPTZ updated_at NOT NULL with the now() server default), and
|
||||||
|
the table is ABSENT at 0016 while the 0016 ``ui_settings`` schema
|
||||||
|
survives the upgrade."""
|
||||||
|
command.downgrade(alembic, "0016") # start from the pre-0017 state
|
||||||
|
assert _version(db) == "0016"
|
||||||
|
assert not _table_exists(db, "folder_summaries"), (
|
||||||
|
"folder_summaries must be absent at 0016"
|
||||||
|
)
|
||||||
|
|
||||||
|
command.upgrade(alembic, "0017")
|
||||||
|
assert _version(db) == "0017", "alembic_version must be at 0017"
|
||||||
|
assert _table_exists(db, "folder_summaries"), "the table must exist at 0017"
|
||||||
|
|
||||||
|
source = _column(db, "folder_summaries", "source")
|
||||||
|
assert source is not None, "folder_summaries.source is missing"
|
||||||
|
assert source[0] == "character varying", "source must be VARCHAR"
|
||||||
|
assert source[1] == "NO", "source must be NOT NULL (PK part 1)"
|
||||||
|
assert source[2] is None, "source must have no server default"
|
||||||
|
assert source[3] == 120, "source must be String(120) — documents.source"
|
||||||
|
|
||||||
|
folder = _column(db, "folder_summaries", "folder_path")
|
||||||
|
assert folder is not None, "folder_summaries.folder_path is missing"
|
||||||
|
assert folder[0] == "character varying", "folder_path must be VARCHAR"
|
||||||
|
assert folder[1] == "NO", "folder_path must be NOT NULL (PK part 2)"
|
||||||
|
assert folder[2] is None, "folder_path must have no server default"
|
||||||
|
assert folder[3] == 1000, "folder_path must be String(1000) — documents.path"
|
||||||
|
|
||||||
|
summary = _column(db, "folder_summaries", "summary")
|
||||||
|
assert summary is not None, "folder_summaries.summary is missing"
|
||||||
|
assert summary[0] == "text", "summary must be TEXT"
|
||||||
|
assert summary[1] == "NO", "summary must be NOT NULL (never stored empty)"
|
||||||
|
|
||||||
|
updated = _column(db, "folder_summaries", "updated_at")
|
||||||
|
assert updated is not None, "folder_summaries.updated_at is missing"
|
||||||
|
assert updated[0] == "timestamp with time zone", "updated_at must be TIMESTAMPTZ"
|
||||||
|
assert updated[1] == "NO", "updated_at must be NOT NULL"
|
||||||
|
assert updated[2] is not None and "now" in str(updated[2]), (
|
||||||
|
"updated_at must carry the now() server default (house style)"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert _pk_columns(db, "folder_summaries") == ["source", "folder_path"]
|
||||||
|
|
||||||
|
# The 0016 schema survives the additive upgrade.
|
||||||
|
semantic = _column(db, "ui_settings", "ok_bg")
|
||||||
|
assert semantic is not None and semantic[3] == 7, (
|
||||||
|
"ui_settings.ok_bg (0016) must survive the upgrade"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_inserted_rows_round_trip(db: Session, alembic: Config) -> None:
|
||||||
|
"""At 0017, a source-root row (``folder_path = ''``) and a nested
|
||||||
|
folder row round-trip their values, and the composite PK rejects a
|
||||||
|
duplicate (source, folder_path) pair."""
|
||||||
|
command.upgrade(alembic, "head")
|
||||||
|
try:
|
||||||
|
db.execute(
|
||||||
|
text(
|
||||||
|
"INSERT INTO folder_summaries (source, folder_path, summary)"
|
||||||
|
" VALUES ('Homelab', '', 'Root summary.')"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
db.execute(
|
||||||
|
text(
|
||||||
|
"INSERT INTO folder_summaries (source, folder_path, summary)"
|
||||||
|
" VALUES ('Homelab', 'deployments/ansible', 'Ansible summary.')"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
rows = db.execute(
|
||||||
|
text(
|
||||||
|
"SELECT source, folder_path, summary, updated_at"
|
||||||
|
" FROM folder_summaries ORDER BY folder_path"
|
||||||
|
)
|
||||||
|
).fetchall()
|
||||||
|
assert len(rows) == 2, "both rows must be stored"
|
||||||
|
assert rows[0][0] == "Homelab" and rows[0][1] == "", (
|
||||||
|
"the source-root row uses folder_path = ''"
|
||||||
|
)
|
||||||
|
assert rows[0][2] == "Root summary.", "summary must round-trip verbatim"
|
||||||
|
assert rows[0][3] is not None, "updated_at must be stamped (server default)"
|
||||||
|
assert rows[1][1] == "deployments/ansible", (
|
||||||
|
"a nested folder path must round-trip verbatim"
|
||||||
|
)
|
||||||
|
assert rows[1][2] == "Ansible summary."
|
||||||
|
|
||||||
|
with pytest.raises(IntegrityError):
|
||||||
|
db.execute(
|
||||||
|
text(
|
||||||
|
"INSERT INTO folder_summaries (source, folder_path, summary)"
|
||||||
|
" VALUES ('Homelab', 'deployments/ansible', 'dup')"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
db.rollback() # the IntegrityError aborts the open transaction
|
||||||
|
finally:
|
||||||
|
_clear_rows(db)
|
||||||
|
|
||||||
|
|
||||||
|
def test_downgrade_to_0016_drops_the_table(db: Session, alembic: Config) -> None:
|
||||||
|
"""Downgrade 0017 → 0016: the table is gone (A13 — fully
|
||||||
|
reversible) while the rest of the schema survives (the 0016
|
||||||
|
``ui_settings`` semantic columns, ``api_tokens``, ``documents``)."""
|
||||||
|
command.downgrade(alembic, "0016")
|
||||||
|
assert _version(db) == "0016"
|
||||||
|
assert not _table_exists(db, "folder_summaries"), (
|
||||||
|
"folder_summaries must be dropped"
|
||||||
|
)
|
||||||
|
|
||||||
|
semantic = _column(db, "ui_settings", "accent_line")
|
||||||
|
assert semantic is not None and semantic[3] == 7, (
|
||||||
|
"ui_settings.accent_line (0016) must survive the downgrade"
|
||||||
|
)
|
||||||
|
token_col = _column(db, "api_tokens", "token_hash")
|
||||||
|
assert token_col is not None and token_col[0] == "character varying", (
|
||||||
|
"api_tokens.token_hash must survive the downgrade"
|
||||||
|
)
|
||||||
|
doc_path = _column(db, "documents", "path")
|
||||||
|
assert doc_path is not None and doc_path[3] == 1000, (
|
||||||
|
"documents.path must survive the downgrade"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_round_trip_restores_the_table(db: Session, alembic: Config) -> None:
|
||||||
|
"""Downgrade to 0016, then upgrade back to 0017: the table is back
|
||||||
|
with the column contract and PK intact."""
|
||||||
|
command.downgrade(alembic, "0016")
|
||||||
|
command.upgrade(alembic, "0017")
|
||||||
|
assert _version(db) == "0017", "round-trip upgrade must land at 0017"
|
||||||
|
assert _table_exists(db, "folder_summaries"), "the table must be back"
|
||||||
|
|
||||||
|
folder = _column(db, "folder_summaries", "folder_path")
|
||||||
|
assert folder is not None, "folder_summaries.folder_path must be back"
|
||||||
|
assert folder[0] == "character varying", "folder_path must be VARCHAR"
|
||||||
|
assert folder[1] == "NO", "folder_path must be NOT NULL after the round-trip"
|
||||||
|
assert folder[3] == 1000, "folder_path must be String(1000) after the round-trip"
|
||||||
|
assert _pk_columns(db, "folder_summaries") == ["source", "folder_path"]
|
||||||
@@ -48,9 +48,16 @@ and every failure path (git error, model down) never bumps. The
|
|||||||
counter is pinned to the migration-0010 seed (0) around every test by
|
counter is pinned to the migration-0010 seed (0) around every test by
|
||||||
:func:`_reset_sources_version`.
|
:func:`_reset_sources_version`.
|
||||||
|
|
||||||
The git / import / overview layers are monkeypatched in ``app.api.sync``
|
The git / import / overview / folder-summary layers are monkeypatched
|
||||||
(same fake style as ``test_import_docs_git.py``) — no real git, no LLM:
|
in ``app.api.sync`` (same fake style as ``test_import_docs_git.py``) —
|
||||||
the runner's state machine and HTTP surface are under test.
|
no real git, no LLM: the runner's state machine and HTTP surface are
|
||||||
|
under test. Phase 94 (task 02): the folder-summary layer gets the same
|
||||||
|
treatment — the fake-import tests' canned summaries would otherwise
|
||||||
|
steer the REAL ``generate_folder_summaries`` at the global
|
||||||
|
``documents`` table and the real ``LLMClient`` (network); the
|
||||||
|
real-import tests (host temp dirs, deterministic ``FakeEmbedder``) keep
|
||||||
|
the real generator, with ``folder_summaries`` truncated around every
|
||||||
|
test (:func:`_clean_folder_summaries`).
|
||||||
|
|
||||||
The admin client is used **as a context manager** on purpose: the
|
The admin client is used **as a context manager** on purpose: the
|
||||||
background sync task lives on the app's event loop, so the loop must
|
background sync task lives on the app's event loop, so the loop must
|
||||||
@@ -182,6 +189,19 @@ def clean_documents(db: Session) -> Iterator[None]:
|
|||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _clean_folder_summaries(db: Session) -> Iterator[None]:
|
||||||
|
"""Phase 94: the ``folder_summaries`` table is global state the real
|
||||||
|
generator (the real-import tests) writes — truncated around every
|
||||||
|
sync test so the change-gate / table-empty-gate assertions start
|
||||||
|
from a known (empty) table."""
|
||||||
|
db.execute(text("TRUNCATE folder_summaries"))
|
||||||
|
db.commit()
|
||||||
|
yield
|
||||||
|
db.execute(text("TRUNCATE folder_summaries"))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
def _real_llm(monkeypatch: pytest.MonkeyPatch) -> None:
|
def _real_llm(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
"""The pipeline's ``LLMClient`` becomes the deterministic in-process
|
"""The pipeline's ``LLMClient`` becomes the deterministic in-process
|
||||||
``FakeEmbedder`` (real import, no network). ``FakeEmbedder``
|
``FakeEmbedder`` (real import, no network). ``FakeEmbedder``
|
||||||
@@ -276,6 +296,34 @@ class FakeOverview:
|
|||||||
return self.ok
|
return self.ok
|
||||||
|
|
||||||
|
|
||||||
|
class FakeFolderSummaries:
|
||||||
|
"""Records every ``generate_folder_summaries`` call; canned stats.
|
||||||
|
|
||||||
|
Phase 94 (task 02): keeps the fake-import tests at the deterministic
|
||||||
|
layer boundary — the real generator would read the global
|
||||||
|
``documents`` table and call the (real) ``LLMClient`` over the
|
||||||
|
network. The generator only flushes, so the fake honours the
|
||||||
|
``skip`` flag the same way (the zero stats, no side effects)."""
|
||||||
|
|
||||||
|
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
|
||||||
|
|
||||||
|
def __init__(self, stats: dict[str, int] | None = None) -> None:
|
||||||
|
self.stats = stats if stats is not None else dict(self.ZERO)
|
||||||
|
self.llms: list[LLMClient] = []
|
||||||
|
self.sessions: list[Session] = []
|
||||||
|
self.skip_flags: list[bool] = []
|
||||||
|
|
||||||
|
async def __call__(
|
||||||
|
self, db: Session, llm: LLMClient, *, skip: bool = False
|
||||||
|
) -> dict[str, int]:
|
||||||
|
self.skip_flags.append(skip)
|
||||||
|
if skip:
|
||||||
|
return dict(self.ZERO)
|
||||||
|
self.llms.append(llm)
|
||||||
|
self.sessions.append(db)
|
||||||
|
return dict(self.stats)
|
||||||
|
|
||||||
|
|
||||||
def _fake_clone() -> tuple[list[tuple[str, Path]], object]:
|
def _fake_clone() -> tuple[list[tuple[str, Path]], object]:
|
||||||
"""A ``clone_or_pull`` that materialises a checkout with one .md file."""
|
"""A ``clone_or_pull`` that materialises a checkout with one .md file."""
|
||||||
calls: list[tuple[str, Path]] = []
|
calls: list[tuple[str, Path]] = []
|
||||||
@@ -327,6 +375,7 @@ def test_admin_sync_success_reports_full_detail(
|
|||||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
fake_overview = FakeOverview(ok=True)
|
fake_overview = FakeOverview(ok=True)
|
||||||
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
||||||
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
||||||
|
|
||||||
_login(sync_client)
|
_login(sync_client)
|
||||||
assert sync_client.get("/api/sync/status").json() == {
|
assert sync_client.get("/api/sync/status").json() == {
|
||||||
@@ -399,6 +448,7 @@ def test_unchanged_kb_skips_overview_refresh(
|
|||||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
fake_overview = FakeOverview(ok=True)
|
fake_overview = FakeOverview(ok=True)
|
||||||
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
||||||
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
||||||
|
|
||||||
_login(sync_client)
|
_login(sync_client)
|
||||||
assert sync_client.post("/api/sync").status_code == 202
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
@@ -435,6 +485,7 @@ def test_double_trigger_while_running_returns_409(
|
|||||||
fake_import = FakeImportSources(ImportSummary(files=1, added=1), delay=0.5)
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1), delay=0.5)
|
||||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||||
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
||||||
|
|
||||||
_login(sync_client)
|
_login(sync_client)
|
||||||
assert sync_client.post("/api/sync").status_code == 202
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
@@ -551,12 +602,17 @@ def test_db_rows_win_over_env(
|
|||||||
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
||||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||||
|
fake_folders = FakeFolderSummaries()
|
||||||
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
|
||||||
|
|
||||||
_login(sync_client)
|
_login(sync_client)
|
||||||
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
||||||
assert sync_client.post("/api/sync").status_code == 202
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
body = _poll(sync_client, "success")
|
body = _poll(sync_client, "success")
|
||||||
|
|
||||||
|
# Phase 94: the changed canned summary fired the (stubbed) folder
|
||||||
|
# step with the run's own session + the import's LLM client.
|
||||||
|
assert fake_folders.llms == [fake_import.llms[0]]
|
||||||
assert clone_calls == [(db_url, tmp_path / "bor" / "managed")]
|
assert clone_calls == [(db_url, tmp_path / "bor" / "managed")]
|
||||||
assert fake_import.sources == [[tmp_path / "bor" / "managed"]]
|
assert fake_import.sources == [[tmp_path / "bor" / "managed"]]
|
||||||
assert "env.example.com" not in str(body) # the env URL never reaches the UI
|
assert "env.example.com" not in str(body) # the env URL never reaches the UI
|
||||||
@@ -584,6 +640,7 @@ def test_env_fallback_when_table_empty(
|
|||||||
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
||||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||||
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
||||||
|
|
||||||
_login(sync_client)
|
_login(sync_client)
|
||||||
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
||||||
@@ -988,6 +1045,7 @@ def test_sync_builds_ignore_map_by_root_string_with_union(
|
|||||||
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
||||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||||
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
||||||
|
|
||||||
_login(sync_client)
|
_login(sync_client)
|
||||||
assert sync_client.post("/api/sync").status_code == 202
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
@@ -1022,6 +1080,7 @@ def test_sync_without_ignore_lists_passes_empty_map(
|
|||||||
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
|
||||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||||
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
||||||
|
|
||||||
_login(sync_client)
|
_login(sync_client)
|
||||||
assert sync_client.post("/api/sync").status_code == 202
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
|||||||
@@ -0,0 +1,664 @@
|
|||||||
|
"""Integration: the sync-time folder-summary wiring (phase 94, task 02).
|
||||||
|
|
||||||
|
Extends the KB-overview sync pattern (``test_import_docs_overview.py``
|
||||||
|
is the template) to the phase-94 folder summaries — both sync paths
|
||||||
|
regenerate ``folder_summaries`` change-gated (added + updated > 0) or
|
||||||
|
on an empty table (the first full run after migration 0017 / a
|
||||||
|
``--limit`` first walk), per-folder fail-soft, best-effort, and in the
|
||||||
|
run's own short-lived session (the phase-53 flush-then-caller-commits
|
||||||
|
convention — the generator only flushes, the sync path commits).
|
||||||
|
|
||||||
|
Script path (``scripts.import_docs.main`` end to end, fake LLM, real
|
||||||
|
DB, explicit ``--source``):
|
||||||
|
|
||||||
|
- a KB-changing import → one row per ≥ 2-doc subtree (the source root
|
||||||
|
+ the 2-doc folder; the 1-doc folder gets none), committed in the
|
||||||
|
run's transaction, the summary line ending
|
||||||
|
``folder_summaries=<generated>/<failed>/<pruned>``;
|
||||||
|
- an unchanged re-import → zero ``lite`` calls,
|
||||||
|
``folder_summaries=skipped``, rows untouched;
|
||||||
|
- a subtree dropping below 2 docs after a changed re-walk → its row
|
||||||
|
pruned;
|
||||||
|
- one folder's ``lite`` failure → its previous row kept, the other
|
||||||
|
lands, exit code 0, stats ``1/1/0``;
|
||||||
|
- a ``--limit`` debug run → no generation, no rows,
|
||||||
|
``folder_summaries=skipped``;
|
||||||
|
- a fresh (empty) table after a ``--limit`` first walk → an unchanged
|
||||||
|
full walk generates (the table-empty first-run trigger).
|
||||||
|
|
||||||
|
API path (``POST /api/sync`` end to end, real import over a host temp
|
||||||
|
local dir, deterministic ``FakeEmbedder``):
|
||||||
|
|
||||||
|
- a KB-changing sync → the rows land (visible via the test's own
|
||||||
|
session) and the status detail keeps its exact pre-phase key set
|
||||||
|
(no folder-summary surface — the stats are log-only);
|
||||||
|
- an unchanged re-sync → zero ``FOLDER_SUMMARY_MODE`` calls;
|
||||||
|
- a ``lite`` outage (one folder failing) → the failed folder's row is
|
||||||
|
kept, the run reports ``success`` (never ``failed``), and the
|
||||||
|
sources-version bump still lands (the bump is change-gated on the
|
||||||
|
KB, not on the summaries);
|
||||||
|
- an empty table after a populated sync (the migration-0017 scenario)
|
||||||
|
→ an unchanged walk regenerates (the overview's API gate, purely
|
||||||
|
change-gated, does not).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import select, text
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.api import sync as sync_api
|
||||||
|
from app.config import Settings
|
||||||
|
from app.db import SessionLocal, db_available
|
||||||
|
from app.main import app as fastapi_app
|
||||||
|
from app.models import FolderSummary, GitSource
|
||||||
|
from app.rag import git_sources as git_sources_resolver
|
||||||
|
from app.rag.llm import LLMError
|
||||||
|
from app.rag.sources_meta import current_sources_version
|
||||||
|
from scripts import import_docs
|
||||||
|
from tests.conftest import ADMIN_PASSWORD
|
||||||
|
from tests.fakes import FakeEmbedder
|
||||||
|
|
||||||
|
# --- shared helpers ---------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _folder_calls(llm: FakeEmbedder) -> list[list[dict[str, str]]]:
|
||||||
|
"""The ``FOLDER_SUMMARY_MODE`` chat calls on the client (the marker
|
||||||
|
in the system prompt — the KB-overview marker never contains it)."""
|
||||||
|
return [
|
||||||
|
msgs
|
||||||
|
for msgs in llm.chat_calls
|
||||||
|
if any(
|
||||||
|
"FOLDER_SUMMARY_MODE" in m.get("content", "")
|
||||||
|
for m in msgs
|
||||||
|
if m.get("role") == "system"
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _rows(db: Session) -> dict[tuple[str, str], str]:
|
||||||
|
"""The stored folder summaries as ``{(source, folder_path): summary}``."""
|
||||||
|
db.expire_all()
|
||||||
|
return {
|
||||||
|
(source, folder_path): summary
|
||||||
|
for source, folder_path, summary in db.execute(
|
||||||
|
select(FolderSummary.source, FolderSummary.folder_path, FolderSummary.summary)
|
||||||
|
).all()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _updated_at(db: Session, source: str, folder_path: str) -> datetime | None:
|
||||||
|
db.expire_all()
|
||||||
|
row = db.get(FolderSummary, (source, folder_path))
|
||||||
|
return row.updated_at if row is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
class FolderFailingChatEmbedder(FakeEmbedder):
|
||||||
|
"""``lite`` stand-in that fails exactly one folder-summary call —
|
||||||
|
the one whose user message contains *fail_label* (a ``Folder: …``
|
||||||
|
header) — and answers everything else (incl. the KB overview)
|
||||||
|
normally. Drives the per-folder fail-soft path."""
|
||||||
|
|
||||||
|
def __init__(self, fail_label: str) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self.fail_label = fail_label
|
||||||
|
|
||||||
|
async def chat(
|
||||||
|
self, messages: list[dict[str, str]], model: str | None = None
|
||||||
|
) -> str:
|
||||||
|
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||||
|
if self.fail_label in user:
|
||||||
|
self.chat_calls.append(list(messages))
|
||||||
|
raise LLMError("simulated folder-summary outage (test sentinel)")
|
||||||
|
return await super().chat(messages, model)
|
||||||
|
|
||||||
|
|
||||||
|
# --- fixtures ----------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _clean_kb(db: Session) -> Iterator[None]:
|
||||||
|
"""Global KB + registry state, truncated around every test (the
|
||||||
|
real import and the real generator write ``documents``/``chunks``/
|
||||||
|
``folder_summaries``; the API tests seed ``git_sources``)."""
|
||||||
|
db.execute(
|
||||||
|
text("TRUNCATE chunks, documents, kb_overview, folder_summaries, git_sources")
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
yield
|
||||||
|
db.execute(
|
||||||
|
text("TRUNCATE chunks, documents, kb_overview, folder_summaries, git_sources")
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _reset_sources_version() -> Iterator[None]:
|
||||||
|
"""The sources version counter is global mutable state — pin it to
|
||||||
|
the migration-0010 seed (0) around every test (own session: both
|
||||||
|
sync paths bump through their own short-lived ``SessionLocal``).
|
||||||
|
Skips like the ``db`` fixture when Postgres is down."""
|
||||||
|
if not db_available():
|
||||||
|
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||||
|
session = SessionLocal()
|
||||||
|
try:
|
||||||
|
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
||||||
|
session.commit()
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
session.execute(text("UPDATE sources_meta SET version = 0 WHERE id = 1"))
|
||||||
|
session.commit()
|
||||||
|
session.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _fresh_sync_state() -> Iterator[None]:
|
||||||
|
"""The module-level status object + task are process-global: reset
|
||||||
|
them around every test (harmless for the script-path tests)."""
|
||||||
|
sync_api._status = sync_api.SyncStatus()
|
||||||
|
sync_api._task = None
|
||||||
|
yield
|
||||||
|
sync_api._status = sync_api.SyncStatus()
|
||||||
|
sync_api._task = None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def src(tmp_path: Path) -> Path:
|
||||||
|
"""MyDocs: a/ (2 docs) + b/ (1 doc) → subtree counts root 3, a 2, b 1.
|
||||||
|
|
||||||
|
md → no document-summary chat calls, so the ``lite`` traffic is
|
||||||
|
exactly the overview + the folder summaries."""
|
||||||
|
root = tmp_path / "MyDocs"
|
||||||
|
(root / "a").mkdir(parents=True)
|
||||||
|
(root / "b").mkdir()
|
||||||
|
(root / "a" / "one.md").write_text("# A One\nFirst folder document.\n", encoding="utf-8")
|
||||||
|
(root / "a" / "two.md").write_text("# A Two\nSecond folder document.\n", encoding="utf-8")
|
||||||
|
(root / "b" / "three.md").write_text("# B Three\nLone folder document.\n", encoding="utf-8")
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
def _run_main(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
llm: FakeEmbedder,
|
||||||
|
argv: list[str],
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> tuple[int, str]:
|
||||||
|
"""Run ``import_docs.main`` with fresh settings, a fake LLM, and a
|
||||||
|
fail-loud git mock (``--source`` always wins, so git must stay
|
||||||
|
idle) — the phase-31 overview test's runner."""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
import_docs,
|
||||||
|
"get_settings",
|
||||||
|
lambda: Settings(_env_file=None), # pyright: ignore[reportCallIssue]
|
||||||
|
)
|
||||||
|
|
||||||
|
def _no_git(url: str, dest: Path | str) -> Path:
|
||||||
|
raise AssertionError("git sync must not run with explicit --source")
|
||||||
|
|
||||||
|
monkeypatch.setattr(import_docs, "clone_or_pull", _no_git)
|
||||||
|
monkeypatch.setattr(import_docs, "LLMClient", lambda: llm)
|
||||||
|
rc = import_docs.main(argv)
|
||||||
|
return rc, capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
# --- script path: scripts/import_docs.py -------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_changed_import_generates_folder_rows(
|
||||||
|
db: Session,
|
||||||
|
src: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
"""A KB-changing import upserts one row per ≥ 2-doc subtree in the
|
||||||
|
run's own transaction — committed and visible afterwards — with the
|
||||||
|
stats on the summary line (PLAN §9)."""
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
records: list[logging.LogRecord] = []
|
||||||
|
|
||||||
|
class _Sink(logging.Handler):
|
||||||
|
def emit(self, record: logging.LogRecord) -> None:
|
||||||
|
records.append(record)
|
||||||
|
|
||||||
|
fs_logger = logging.getLogger("app.rag.folder_summaries")
|
||||||
|
sink = _Sink()
|
||||||
|
fs_logger.addHandler(sink)
|
||||||
|
fs_logger.setLevel(logging.INFO)
|
||||||
|
try:
|
||||||
|
rc, out = _run_main(monkeypatch, llm, ["--source", str(src)], capsys)
|
||||||
|
finally:
|
||||||
|
fs_logger.removeHandler(sink)
|
||||||
|
|
||||||
|
assert rc == 0
|
||||||
|
assert "added=3" in out
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=updated sources_version=1 folder_summaries=2/0/0"
|
||||||
|
)
|
||||||
|
# Two FOLDER_SUMMARY_MODE calls — the source root (3 docs) and the
|
||||||
|
# 2-doc folder a; the 1-doc folder b gets no row (no lite burn).
|
||||||
|
calls = _folder_calls(llm)
|
||||||
|
assert len(calls) == 2
|
||||||
|
headers = [c[1]["content"].splitlines()[0] for c in calls]
|
||||||
|
assert headers == ["Folder: MyDocs", "Folder: MyDocs/a"]
|
||||||
|
for c in calls:
|
||||||
|
assert "FOLDER_SUMMARY_MODE" in c[0]["content"]
|
||||||
|
# The rows land committed (the run's own session committed them) —
|
||||||
|
# one per ≥ 2-doc subtree, never empty, stamped.
|
||||||
|
rows = _rows(db)
|
||||||
|
assert set(rows) == {("MyDocs", ""), ("MyDocs", "a")}
|
||||||
|
assert all(rows.values())
|
||||||
|
assert _updated_at(db, "MyDocs", "a") is not None
|
||||||
|
# The stats log line (PLAN §9 ample logging).
|
||||||
|
assert any(
|
||||||
|
"folder_summaries: generated=2 failed=0 pruned=0" in r.getMessage()
|
||||||
|
for r in records
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_unchanged_reimport_burns_zero_folder_calls(
|
||||||
|
db: Session,
|
||||||
|
src: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
"""Same hashes → no KB change → the populated table stays untouched
|
||||||
|
and zero ``lite`` calls burn (overview AND folder summaries)."""
|
||||||
|
llm1 = FakeEmbedder()
|
||||||
|
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
|
||||||
|
assert rc == 0
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=updated sources_version=1 folder_summaries=2/0/0"
|
||||||
|
)
|
||||||
|
rows = _rows(db)
|
||||||
|
assert set(rows) == {("MyDocs", ""), ("MyDocs", "a")}
|
||||||
|
|
||||||
|
llm2 = FakeEmbedder()
|
||||||
|
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||||
|
assert rc == 0
|
||||||
|
assert "unchanged=3" in out
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=skipped sources_version=skipped folder_summaries=skipped"
|
||||||
|
)
|
||||||
|
assert llm2.chat_calls == [] # zero lite calls, any mode
|
||||||
|
assert _rows(db) == rows # rows byte-identical
|
||||||
|
|
||||||
|
|
||||||
|
def test_subtree_dropping_below_two_docs_is_pruned(
|
||||||
|
db: Session,
|
||||||
|
src: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
"""A subtree that drops below the ≥ 2 rule on a changed re-walk
|
||||||
|
loses its (stale) row; the remaining qualifiers regenerate."""
|
||||||
|
llm1 = FakeEmbedder()
|
||||||
|
rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
|
||||||
|
assert rc == 0
|
||||||
|
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")}
|
||||||
|
|
||||||
|
# a/ loses one of its two docs (below the ≥ 2 rule) while b's doc
|
||||||
|
# changes → the re-walk is a KB change, so generation runs and the
|
||||||
|
# stale a/ row is pruned; b (1 doc) still gets no row.
|
||||||
|
(src / "a" / "two.md").unlink()
|
||||||
|
(src / "b" / "three.md").write_text("# B Three\nChanged content.\n", encoding="utf-8")
|
||||||
|
llm2 = FakeEmbedder()
|
||||||
|
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src), "--prune"], capsys)
|
||||||
|
|
||||||
|
assert rc == 0
|
||||||
|
assert "pruned=1" in out
|
||||||
|
assert "updated=1" in out
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=updated sources_version=2 folder_summaries=1/0/1"
|
||||||
|
)
|
||||||
|
# Only the source root (the 2 remaining docs) still qualifies.
|
||||||
|
assert set(_rows(db)) == {("MyDocs", "")}
|
||||||
|
assert _updated_at(db, "MyDocs", "a") is None # the row is gone
|
||||||
|
|
||||||
|
|
||||||
|
def test_folder_lite_failure_keeps_previous_row_and_stays_green(
|
||||||
|
db: Session,
|
||||||
|
src: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
"""One folder's ``lite`` failure: its previous row is kept, the
|
||||||
|
other folder lands, and the run's exit code stays 0 — the stats
|
||||||
|
carry the failure (``1/1/0``)."""
|
||||||
|
llm1 = FakeEmbedder()
|
||||||
|
rc, _ = _run_main(monkeypatch, llm1, ["--source", str(src)], capsys)
|
||||||
|
assert rc == 0
|
||||||
|
rows_before = _rows(db)
|
||||||
|
a_stamp_before = _updated_at(db, "MyDocs", "a")
|
||||||
|
root_stamp_before = _updated_at(db, "MyDocs", "")
|
||||||
|
assert a_stamp_before is not None and root_stamp_before is not None
|
||||||
|
|
||||||
|
# a/ changes (a KB change → generation runs); lite fails for a/
|
||||||
|
# only — its previous row must stay, the root row must land.
|
||||||
|
(src / "a" / "one.md").write_text("# A One\nChanged content.\n", encoding="utf-8")
|
||||||
|
llm2 = FolderFailingChatEmbedder(fail_label="Folder: MyDocs/a")
|
||||||
|
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||||
|
|
||||||
|
assert rc == 0 # a failed folder must not fail the import
|
||||||
|
assert "updated=1" in out
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=updated sources_version=2 folder_summaries=1/1/0"
|
||||||
|
)
|
||||||
|
rows_after = _rows(db)
|
||||||
|
assert rows_after["MyDocs", "a"] == rows_before["MyDocs", "a"] # kept
|
||||||
|
assert _updated_at(db, "MyDocs", "a") == a_stamp_before # untouched
|
||||||
|
root_stamp_after = _updated_at(db, "MyDocs", "")
|
||||||
|
assert root_stamp_after is not None and root_stamp_after > root_stamp_before
|
||||||
|
|
||||||
|
|
||||||
|
def test_limit_run_skips_folder_generation(
|
||||||
|
db: Session,
|
||||||
|
src: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
"""A ``--limit`` debug run (an incomplete walk) never generates —
|
||||||
|
no ``lite`` call at all, no rows, the line says ``skipped``."""
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
rc, out = _run_main(monkeypatch, llm, ["--source", str(src), "--limit", "2"], capsys)
|
||||||
|
assert rc == 0
|
||||||
|
assert "added=2" in out
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=skipped sources_version=skipped folder_summaries=skipped"
|
||||||
|
)
|
||||||
|
assert llm.chat_calls == [] # no lite call, any mode
|
||||||
|
assert _rows(db) == {} # an incomplete walk never writes rows
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_table_generates_on_unchanged_walk(
|
||||||
|
db: Session,
|
||||||
|
src: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
capsys: pytest.CaptureFixture[str],
|
||||||
|
) -> None:
|
||||||
|
"""The table-empty first-run trigger: after a ``--limit`` first
|
||||||
|
walk (populated KB, empty table), an unchanged full walk generates
|
||||||
|
— for the folder summaries AND the missing outline, still never
|
||||||
|
bumping the version."""
|
||||||
|
llm1 = FakeEmbedder()
|
||||||
|
rc, out = _run_main(monkeypatch, llm1, ["--source", str(src), "--limit", "3"], capsys)
|
||||||
|
assert rc == 0
|
||||||
|
assert "added=3" in out
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=skipped sources_version=skipped folder_summaries=skipped"
|
||||||
|
)
|
||||||
|
assert _rows(db) == {}
|
||||||
|
|
||||||
|
llm2 = FakeEmbedder()
|
||||||
|
rc, out = _run_main(monkeypatch, llm2, ["--source", str(src)], capsys)
|
||||||
|
assert rc == 0
|
||||||
|
assert "unchanged=3" in out
|
||||||
|
assert out.rstrip().endswith(
|
||||||
|
"overview=updated sources_version=skipped folder_summaries=2/0/0"
|
||||||
|
)
|
||||||
|
assert set(_rows(db)) == {("MyDocs", ""), ("MyDocs", "a")}
|
||||||
|
assert len(_folder_calls(llm2)) == 2
|
||||||
|
assert current_sources_version(db) == 0 # the unchanged walk never bumps
|
||||||
|
|
||||||
|
|
||||||
|
# --- API path: POST /api/sync -------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def sync_client() -> Iterator[TestClient]:
|
||||||
|
"""Context-managed TestClient — one app event loop across requests
|
||||||
|
(the background task must survive between the POST and the polls)."""
|
||||||
|
with TestClient(fastapi_app) as client:
|
||||||
|
yield client
|
||||||
|
|
||||||
|
|
||||||
|
def _settings(sources_dir: str) -> Settings:
|
||||||
|
return Settings(_env_file=None, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
||||||
|
|
||||||
|
|
||||||
|
def _stub_env(monkeypatch: pytest.MonkeyPatch, git_sources: str = "") -> None:
|
||||||
|
"""The resolver's env fallback, driven by a fresh ``Settings`` (the
|
||||||
|
dev ``.env`` never leaks in)."""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
git_sources_resolver,
|
||||||
|
"get_settings",
|
||||||
|
lambda: Settings(_env_file=None, git_sources=git_sources), # pyright: ignore[reportCallIssue]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_local(db: Session, path: Path) -> None:
|
||||||
|
"""A ``kind=local`` row as the phase-38 API stores it: the expanded
|
||||||
|
absolute path in both ``path`` and the NOT-NULL ``url`` column."""
|
||||||
|
db.add(GitSource(url=str(path), kind="local", path=str(path)))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_llm(monkeypatch: pytest.MonkeyPatch) -> list[FakeEmbedder]:
|
||||||
|
"""The pipeline's ``LLMClient`` becomes a recorded
|
||||||
|
``FakeEmbedder`` (real import + real generator, no network)."""
|
||||||
|
clients: list[FakeEmbedder] = []
|
||||||
|
|
||||||
|
def _factory() -> FakeEmbedder:
|
||||||
|
client = FakeEmbedder()
|
||||||
|
clients.append(client)
|
||||||
|
return client
|
||||||
|
|
||||||
|
monkeypatch.setattr(sync_api, "LLMClient", _factory)
|
||||||
|
return clients
|
||||||
|
|
||||||
|
|
||||||
|
def _login(client: TestClient) -> None:
|
||||||
|
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||||
|
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
|
||||||
|
|
||||||
|
|
||||||
|
def _poll(client: TestClient, want: str, timeout: float = 10.0) -> dict:
|
||||||
|
"""Poll ``GET /api/sync/status`` until ``state == want`` (terminal).
|
||||||
|
|
||||||
|
Any state other than ``running`` before the deadline fails loudly —
|
||||||
|
an unexpected ``failed`` must never be masked by the wait."""
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
body = client.get("/api/sync/status").json()
|
||||||
|
if body["state"] == want:
|
||||||
|
return body
|
||||||
|
assert body["state"] == "running", (
|
||||||
|
f"unexpected state {body['state']!r} while waiting for {want!r}: {body}"
|
||||||
|
)
|
||||||
|
time.sleep(0.05)
|
||||||
|
raise AssertionError(f"sync did not reach {want!r} within {timeout}s")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def local_dir(tmp_path: Path) -> Path:
|
||||||
|
"""LocalDocs: a/ (2 docs) → subtree counts root 2, a 2 (one
|
||||||
|
qualifying source-root row + one folder row)."""
|
||||||
|
d = tmp_path / "LocalDocs"
|
||||||
|
(d / "a").mkdir(parents=True)
|
||||||
|
(d / "a" / "one.md").write_text("# A One\nfirst local file\n", encoding="utf-8")
|
||||||
|
(d / "a" / "two.md").write_text("# A Two\nsecond local file\n", encoding="utf-8")
|
||||||
|
return d
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_changed_sync_generates_folder_rows(
|
||||||
|
sync_client: TestClient,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
db: Session,
|
||||||
|
local_dir: Path,
|
||||||
|
) -> None:
|
||||||
|
"""A KB-changing sync upserts the rows (visible via the test's own
|
||||||
|
session) and keeps the status detail's exact pre-phase shape — the
|
||||||
|
folder stats are log-only, not a new status surface."""
|
||||||
|
_seed_local(db, local_dir)
|
||||||
|
_stub_env(monkeypatch)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
sync_api,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings(str(local_dir.parent / "bor")),
|
||||||
|
)
|
||||||
|
clients = _capture_llm(monkeypatch)
|
||||||
|
|
||||||
|
_login(sync_client)
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
body = _poll(sync_client, "success")
|
||||||
|
|
||||||
|
assert body["error"] is None
|
||||||
|
# No new sync-status surface: the detail keeps its exact key set.
|
||||||
|
assert set(body["detail"]) == {
|
||||||
|
"files", "added", "updated", "unchanged", "pruned", "errors",
|
||||||
|
"chunks", "summaries", "summary_errors", "overview",
|
||||||
|
"sources_version",
|
||||||
|
}
|
||||||
|
assert body["detail"]["files"] == 2
|
||||||
|
assert body["detail"]["added"] == 2
|
||||||
|
assert body["detail"]["overview"] is True
|
||||||
|
assert body["detail"]["sources_version"] == 1
|
||||||
|
# One LLM client for probe + import + overview + folder summaries;
|
||||||
|
# exactly two FOLDER_SUMMARY_MODE calls (root + the 2-doc a/ folder).
|
||||||
|
assert len(clients) == 1
|
||||||
|
calls = _folder_calls(clients[0])
|
||||||
|
assert len(calls) == 2
|
||||||
|
headers = [c[1]["content"].splitlines()[0] for c in calls]
|
||||||
|
assert headers == ["Folder: LocalDocs", "Folder: LocalDocs/a"]
|
||||||
|
# The rows land committed — visible from the test's own session.
|
||||||
|
rows = _rows(db)
|
||||||
|
assert set(rows) == {("LocalDocs", ""), ("LocalDocs", "a")}
|
||||||
|
assert all(rows.values())
|
||||||
|
assert current_sources_version(db) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_unchanged_resync_burns_zero_folder_calls(
|
||||||
|
sync_client: TestClient,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
db: Session,
|
||||||
|
local_dir: Path,
|
||||||
|
) -> None:
|
||||||
|
"""Same files → unchanged re-sync: no ``lite`` call of any kind
|
||||||
|
(populated table), rows untouched, version unadvanced."""
|
||||||
|
_seed_local(db, local_dir)
|
||||||
|
_stub_env(monkeypatch)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
sync_api,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings(str(local_dir.parent / "bor")),
|
||||||
|
)
|
||||||
|
clients = _capture_llm(monkeypatch)
|
||||||
|
|
||||||
|
_login(sync_client)
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
_poll(sync_client, "success")
|
||||||
|
rows = _rows(db)
|
||||||
|
assert set(rows) == {("LocalDocs", ""), ("LocalDocs", "a")}
|
||||||
|
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
body = _poll(sync_client, "success")
|
||||||
|
assert body["detail"]["overview"] is False
|
||||||
|
assert body["detail"]["sources_version"] == 1 # unchanged → no bump
|
||||||
|
assert len(clients) == 2
|
||||||
|
# Zero GENERATION calls — no folder summary, no KB overview. The
|
||||||
|
# only chat the re-sync's client makes is the phase-41 probe's ping.
|
||||||
|
assert _folder_calls(clients[1]) == []
|
||||||
|
assert all(
|
||||||
|
"KB_OVERVIEW_MODE" not in m.get("content", "")
|
||||||
|
for msgs in clients[1].chat_calls
|
||||||
|
for m in msgs
|
||||||
|
if m.get("role") == "system"
|
||||||
|
)
|
||||||
|
assert len(clients[1].chat_calls) == 1
|
||||||
|
assert clients[1].chat_calls[0][0]["content"] == "ping"
|
||||||
|
assert _rows(db) == rows
|
||||||
|
assert current_sources_version(db) == 1
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_folder_lite_failure_keeps_rows_stays_green_and_bumps(
|
||||||
|
sync_client: TestClient,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
db: Session,
|
||||||
|
local_dir: Path,
|
||||||
|
) -> None:
|
||||||
|
"""A ``lite`` outage for one folder: the run stays ``success``
|
||||||
|
(never ``failed``), the failed folder's previous row is kept, the
|
||||||
|
rest regenerates, and the sources-version bump still lands (the
|
||||||
|
bump is change-gated on the KB, not on the summaries)."""
|
||||||
|
_seed_local(db, local_dir)
|
||||||
|
_stub_env(monkeypatch)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
sync_api,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings(str(local_dir.parent / "bor")),
|
||||||
|
)
|
||||||
|
clients = _capture_llm(monkeypatch)
|
||||||
|
|
||||||
|
_login(sync_client)
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
_poll(sync_client, "success")
|
||||||
|
rows_before = _rows(db)
|
||||||
|
a_stamp_before = _updated_at(db, "LocalDocs", "a")
|
||||||
|
root_stamp_before = _updated_at(db, "LocalDocs", "")
|
||||||
|
assert a_stamp_before is not None and root_stamp_before is not None
|
||||||
|
assert len(clients) == 1 # sync 1 used the recording factory
|
||||||
|
|
||||||
|
# a/ changes (the KB changes) while lite fails for a/ only.
|
||||||
|
(local_dir / "a" / "one.md").write_text(
|
||||||
|
"# A One\nchanged local file\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
sync_api,
|
||||||
|
"LLMClient",
|
||||||
|
lambda: FolderFailingChatEmbedder(fail_label="Folder: LocalDocs/a"),
|
||||||
|
)
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
body = _poll(sync_client, "success") # GREEN — a lite outage is fail-soft
|
||||||
|
|
||||||
|
assert body["error"] is None
|
||||||
|
assert body["detail"]["updated"] == 1
|
||||||
|
assert body["detail"]["sources_version"] == 2 # the bump was not blocked
|
||||||
|
assert current_sources_version(db) == 2
|
||||||
|
rows_after = _rows(db)
|
||||||
|
assert rows_after["LocalDocs", "a"] == rows_before["LocalDocs", "a"]
|
||||||
|
assert _updated_at(db, "LocalDocs", "a") == a_stamp_before # kept as-is
|
||||||
|
root_stamp_after = _updated_at(db, "LocalDocs", "")
|
||||||
|
assert root_stamp_after is not None and root_stamp_after > root_stamp_before
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_empty_table_first_sync_regenerates(
|
||||||
|
sync_client: TestClient,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
db: Session,
|
||||||
|
local_dir: Path,
|
||||||
|
) -> None:
|
||||||
|
"""The migration-0017 scenario: the KB predates the table — wipe
|
||||||
|
the rows and re-sync an unchanged KB: the empty-table trigger
|
||||||
|
fires for the folder summaries (the overview's API gate, purely
|
||||||
|
change-gated, does not)."""
|
||||||
|
_seed_local(db, local_dir)
|
||||||
|
_stub_env(monkeypatch)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
sync_api,
|
||||||
|
"get_settings",
|
||||||
|
lambda: _settings(str(local_dir.parent / "bor")),
|
||||||
|
)
|
||||||
|
clients = _capture_llm(monkeypatch)
|
||||||
|
|
||||||
|
_login(sync_client)
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
_poll(sync_client, "success")
|
||||||
|
assert set(_rows(db)) == {("LocalDocs", ""), ("LocalDocs", "a")}
|
||||||
|
|
||||||
|
db.execute(text("TRUNCATE folder_summaries"))
|
||||||
|
db.commit()
|
||||||
|
assert sync_client.post("/api/sync").status_code == 202
|
||||||
|
body = _poll(sync_client, "success")
|
||||||
|
|
||||||
|
assert body["detail"]["overview"] is False # unchanged → no overview
|
||||||
|
assert body["detail"]["sources_version"] == 1 # unchanged → no bump
|
||||||
|
assert len(clients) == 2
|
||||||
|
assert len(_folder_calls(clients[1])) == 2 # the folders regenerated
|
||||||
|
assert set(_rows(db)) == {("LocalDocs", ""), ("LocalDocs", "a")}
|
||||||
+480
-128
@@ -2,15 +2,20 @@
|
|||||||
harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
|
harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
|
||||||
|
|
||||||
A scripted fake LLM (canned stream sequences) + monkeypatched
|
A scripted fake LLM (canned stream sequences) + monkeypatched
|
||||||
``list_catalog`` / ``list_source_names`` / ``find_document`` /
|
``ls_top`` / ``ls_folder`` / ``list_source_names`` / ``find_document`` /
|
||||||
``all_documents`` — no database, no network. Covers the loop mechanics:
|
``all_documents`` — no database, no network. Covers the loop mechanics:
|
||||||
the ls → read (combined ``source/path``) → answer happy path (event
|
the ls → read (combined ``source/path``) → answer happy path (event
|
||||||
order, holder state, the tools staying offered on every request —
|
order, holder state, the tools staying offered on every request —
|
||||||
phase 45 removed the per-tool budgets, the assistant/tool message
|
phase 45 removed the per-tool budgets, the assistant/tool message
|
||||||
history), the ``ls`` scoping (no-arg full catalog in the phase-63
|
history), the phase-94 drill-down ``ls`` (no-arg top level = the
|
||||||
labeled-field format, a one-source scope, a known source with 0
|
registered sources with counts + stored summaries in the pinned
|
||||||
documents → ``0 documents:`` counted, an unknown-source refusal that
|
``{N} sources:`` template, a source scope = its root folder —
|
||||||
counts nothing), ``read`` on the canonical combined form (split at the
|
subfolders + capped file lines in the pinned folder template —, a
|
||||||
|
``source/folder`` scope = one level deeper, a registered source with 0
|
||||||
|
documents → the ``… — 0 documents, 0 folders:`` header counted, an
|
||||||
|
unknown-source refusal that counts nothing, the NOT-A-FOLDER teaching
|
||||||
|
with the parent's subfolders, the 50-file cap + grep-pointer note),
|
||||||
|
``read`` on the canonical combined form (split at the
|
||||||
FIRST slash, full content, the bare-source-name refusal, the
|
FIRST slash, full content, the bare-source-name refusal, the
|
||||||
already-in-context dedupe, missing-args refusals), the phase-68 ``grep``
|
already-in-context dedupe, missing-args refusals), the phase-68 ``grep``
|
||||||
contract under its new name (the locked A5 pins: fixed substring,
|
contract under its new name (the locked A5 pins: fixed substring,
|
||||||
@@ -149,28 +154,35 @@ def test_agent_tools_names_and_parameters() -> None:
|
|||||||
# Task 05 (live gate iteration 2): the one-call-at-a-time discipline
|
# Task 05 (live gate iteration 2): the one-call-at-a-time discipline
|
||||||
# clause (the harness prior batches calls; the loop executes one
|
# clause (the harness prior batches calls; the loop executes one
|
||||||
# per round — the extras count as unexecuted in the gate).
|
# per round — the extras count as unexecuted in the gate).
|
||||||
|
# Phase 94 (task 03): the description is the drill-down tree
|
||||||
|
# contract (pinned copy — the tool-surface revision, owner
|
||||||
|
# permission 2026-09-10, ``TODO.md`` L4): one level per call,
|
||||||
|
# sources at the top, folders + files below, the file-line format
|
||||||
|
# and the combined-identity handoff to read/grep intact.
|
||||||
assert ls["description"] == (
|
assert ls["description"] == (
|
||||||
"List the indexed documents as `source: X | path: Y | "
|
"List the knowledge base as a tree, one level at a time. "
|
||||||
"title: Z` lines. Call one tool at a time — wait for "
|
"With no path: the synced sources — each with its document "
|
||||||
"this result before your next call."
|
"count and a summary of its contents. With a source name (no "
|
||||||
|
"'/'): that source's top-level folders and files. With a "
|
||||||
|
"`source/folder` path: that folder's subfolders and files. "
|
||||||
|
"Folder lines carry a summary of what the folder contains. "
|
||||||
|
"File lines are `source: X | path: Y | title: Z` — use the "
|
||||||
|
"combined `source/path` with `read` and `grep`. Call one tool "
|
||||||
|
"at a time — wait for this result before your next call."
|
||||||
)
|
)
|
||||||
ls_params = ls["parameters"]
|
ls_params = ls["parameters"]
|
||||||
assert ls_params["type"] == "object"
|
assert ls_params["type"] == "object"
|
||||||
assert ls_params["required"] == [] # path is optional
|
assert ls_params["required"] == [] # path is optional
|
||||||
assert set(ls_params["properties"]) == {"path"}
|
assert set(ls_params["properties"]) == {"path"}
|
||||||
assert ls_params["properties"]["path"]["type"] == "string"
|
assert ls_params["properties"]["path"]["type"] == "string"
|
||||||
# Phase 72: the description states the contract up front — the
|
# Phase 94 (task 03): the 'path' argument teaches the drill-down
|
||||||
# 'path' argument is a source name, not a file or directory path.
|
# semantics — a source name lists that source's top level, a
|
||||||
# Task 05 (live gate iteration 5): the cross-tool contrast clause
|
# `source/folder` path drills one level deeper, omitted lists
|
||||||
# (ls is the ONLY tool whose path is a source name — the model
|
# every source (pinned copy).
|
||||||
# kept transferring that scope to grep's document identity).
|
|
||||||
assert ls_params["properties"]["path"]["description"] == (
|
assert ls_params["properties"]["path"]["description"] == (
|
||||||
"Source name to list one source's documents (e.g. 'homelab') — "
|
"Optional — a source name (e.g. 'homelab') to list its top "
|
||||||
"a source name, not a file or directory path; omit to list "
|
"level, or a `source/folder` path to drill down (e.g. "
|
||||||
"every document. This is the only tool "
|
"'homelab/active'). Omit it to list every source."
|
||||||
"whose `path` is a source name — for "
|
|
||||||
"`read` and `grep` it must be a document's "
|
|
||||||
"combined `source/path`."
|
|
||||||
)
|
)
|
||||||
read = by_name["read"]["function"]
|
read = by_name["read"]["function"]
|
||||||
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
|
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
|
||||||
@@ -285,16 +297,11 @@ def test_refusal_constants_are_harness_aligned() -> None:
|
|||||||
assert agent.UNKNOWN_TOOL == "Unknown tool."
|
assert agent.UNKNOWN_TOOL == "Unknown tool."
|
||||||
assert agent.MISSING_READ_ARGS == "read requires a string argument 'path'."
|
assert agent.MISSING_READ_ARGS == "read requires a string argument 'path'."
|
||||||
assert agent.MISSING_SEARCH_ARGS == "grep requires a string argument 'pattern'."
|
assert agent.MISSING_SEARCH_ARGS == "grep requires a string argument 'pattern'."
|
||||||
# Phase 72: the ls teaching-refusal templates, pinned byte-for-byte
|
# Phase 94 (task 03): the phase-72 document-path teaching refusal is
|
||||||
# (task 01 — the read/grep suggestion templates below, task 02).
|
# DELETED (a ``/`` now names a folder — the drill-down contract);
|
||||||
assert agent.LS_PATH_NOT_A_SOURCE == (
|
# the no-source refusal stays byte-identical (the task's "existing
|
||||||
"'{path}' looks like a document path, not a source name. The "
|
# refusal, teaching parenthetical intact" pin).
|
||||||
"'path' argument of ls filters by source name (e.g. 'homelab') — "
|
assert not hasattr(agent, "LS_PATH_NOT_A_SOURCE")
|
||||||
"omit it to list every document, or read a document by its "
|
|
||||||
"combined 'source/path' string."
|
|
||||||
)
|
|
||||||
# The pre-phase-72 no-source line is the byte-identical prefix of
|
|
||||||
# the extended line — only the teaching parenthetical was appended.
|
|
||||||
assert agent.NO_SOURCE_NOT_A_DIRECTORY.startswith(
|
assert agent.NO_SOURCE_NOT_A_DIRECTORY.startswith(
|
||||||
"No source named '{scope}' — check the ls output."
|
"No source named '{scope}' — check the ls output."
|
||||||
)
|
)
|
||||||
@@ -303,6 +310,11 @@ def test_refusal_constants_are_harness_aligned() -> None:
|
|||||||
"argument is a source name, not a directory — omit it to list "
|
"argument is a source name, not a directory — omit it to list "
|
||||||
"every document.)"
|
"every document.)"
|
||||||
)
|
)
|
||||||
|
# Phase 94 (task 03): the NOT-A-FOLDER drill-down teaching template,
|
||||||
|
# pinned byte-for-byte (argument echoed, parent's subfolders
|
||||||
|
# listed), and the pinned file-line cap constant.
|
||||||
|
assert agent.NOT_A_FOLDER == "'{arg}' is not a folder — {parent} has: {subfolders}"
|
||||||
|
assert agent.LS_MAX_FILE_LINES == 50
|
||||||
# Phase 72 (task 02): the read/grep "did you mean …?" suggestion
|
# Phase 72 (task 02): the read/grep "did you mean …?" suggestion
|
||||||
# templates, pinned byte-for-byte, and the suggestion cap.
|
# templates, pinned byte-for-byte, and the suggestion cap.
|
||||||
assert agent.NO_DOCUMENT_DID_YOU_MEAN == (
|
assert agent.NO_DOCUMENT_DID_YOU_MEAN == (
|
||||||
@@ -348,11 +360,12 @@ def test_list_source_names_empty_registry(monkeypatch: pytest.MonkeyPatch) -> No
|
|||||||
|
|
||||||
|
|
||||||
def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
catalog = [
|
# Phase 94 (task 03): the no-arg ``ls`` is the drill-down TOP level
|
||||||
("Deployments", "backups.md", "Backup Strategy"),
|
# (the registered sources, registry order) — monkeypatched the way
|
||||||
("Homelab", "aws-route53.md", "AWS Route53 Records"),
|
# the phase-70 full-catalog listing used to be.
|
||||||
]
|
monkeypatch.setattr(
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
|
agent, "ls_top", lambda db: [("Deployments", 1, None), ("Homelab", 1, None)]
|
||||||
|
)
|
||||||
target = _doc("Homelab", "aws-route53.md", "AWS Route53 Records", "R53-CONTENT")
|
target = _doc("Homelab", "aws-route53.md", "AWS Route53 Records", "R53-CONTENT")
|
||||||
calls: list[tuple[str, str]] = []
|
calls: list[tuple[str, str]] = []
|
||||||
|
|
||||||
@@ -423,11 +436,7 @@ def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
|
|||||||
assert msgs[3] == {
|
assert msgs[3] == {
|
||||||
"role": "tool",
|
"role": "tool",
|
||||||
"tool_call_id": "call_1",
|
"tool_call_id": "call_1",
|
||||||
"content": (
|
"content": "2 sources:\n\nDeployments — 1 documents\nHomelab — 1 documents",
|
||||||
"2 documents:\n"
|
|
||||||
"source: Deployments | path: backups.md | title: Backup Strategy\n"
|
|
||||||
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
|
|
||||||
),
|
|
||||||
}
|
}
|
||||||
# The second follow-up request carries the read call + the FULL text.
|
# The second follow-up request carries the read call + the FULL text.
|
||||||
msgs = llm.requests[2][0]
|
msgs = llm.requests[2][0]
|
||||||
@@ -448,7 +457,7 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
|
|||||||
) -> None:
|
) -> None:
|
||||||
"""Rare stream with content AND a tool call: the content stays (it was
|
"""Rare stream with content AND a tool call: the content stays (it was
|
||||||
already emitted) and the tool still runs."""
|
already emitted) and the tool still runs."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[
|
[
|
||||||
@@ -460,7 +469,7 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
|
|||||||
pieces = asyncio.run(_run(llm, holder, _settings()))
|
pieces = asyncio.run(_run(llm, holder, _settings()))
|
||||||
assert [type(p) for p in pieces] == [StreamPiece, ToolCallPiece, StreamPiece]
|
assert [type(p) for p in pieces] == [StreamPiece, ToolCallPiece, StreamPiece]
|
||||||
assert holder.tool_calls == 1 # the tool ran despite the content
|
assert holder.tool_calls == 1 # the tool ran despite the content
|
||||||
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
|
assert llm.requests[1][0][3]["content"] == "0 sources:"
|
||||||
|
|
||||||
|
|
||||||
# ---------- phase 74: client history between system and user ----------
|
# ---------- phase 74: client history between system and user ----------
|
||||||
@@ -519,7 +528,7 @@ def test_run_agent_history_survives_a_tool_round(
|
|||||||
"""The tool rounds append assistant/tool messages to the SAME
|
"""The tool rounds append assistant/tool messages to the SAME
|
||||||
``messages`` list — the prior history stays in place between the
|
``messages`` list — the prior history stays in place between the
|
||||||
system prompt and the current question on the SECOND request too."""
|
system prompt and the current question on the SECOND request too."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [])
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
||||||
[StreamPiece("content", "the answer")],
|
[StreamPiece("content", "the answer")],
|
||||||
@@ -534,22 +543,25 @@ def test_run_agent_history_survives_a_tool_round(
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# ---------- ls: full catalog + scoping ----------
|
# ---------- ls: the drill-down tree (phase 94, task 03) ----------
|
||||||
|
|
||||||
|
|
||||||
def test_ls_full_catalog_format(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_ls_top_level_lists_sources_with_summaries(
|
||||||
"""No argument: the full catalog in the phase-63 labeled-field format
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
(``source: X | path: Y | title: Z``) — counted; no registry lookup."""
|
) -> None:
|
||||||
catalog = [
|
"""No argument: the TOP level — the registered sources, registry
|
||||||
("Deployments", "backups.md", "Backup Strategy"),
|
order, each ``{source} — {n} documents`` + the indented summary line
|
||||||
("Homelab", "aws-route53.md", "AWS Route53 Records"),
|
only when stored — the pinned template, counted; the registry IS
|
||||||
]
|
consulted (unlike the phase-70 full catalog, the top level is the
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
|
registry itself)."""
|
||||||
|
monkeypatch.setattr(
|
||||||
def _boom_sources(*_a: Any, **_k: Any) -> None:
|
agent,
|
||||||
raise AssertionError("no registry lookup for an unscoped ls")
|
"ls_top",
|
||||||
|
lambda db: [
|
||||||
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
|
("Deployments", 0, None),
|
||||||
|
("Homelab", 1, "The homelab notes."),
|
||||||
|
],
|
||||||
|
)
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
||||||
@@ -557,22 +569,27 @@ def test_ls_full_catalog_format(monkeypatch: pytest.MonkeyPatch) -> None:
|
|||||||
)
|
)
|
||||||
asyncio.run(_run(llm, holder, _settings()))
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
assert llm.requests[1][0][3]["content"] == (
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
"2 documents:\n"
|
"2 sources:\n"
|
||||||
"source: Deployments | path: backups.md | title: Backup Strategy\n"
|
"\n"
|
||||||
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
|
"Deployments — 0 documents\n"
|
||||||
|
"Homelab — 1 documents\n"
|
||||||
|
" The homelab notes."
|
||||||
)
|
)
|
||||||
assert holder.tool_calls == 1
|
assert holder.tool_calls == 1
|
||||||
|
|
||||||
|
|
||||||
def test_ls_empty_catalog_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_ls_empty_registry_says_zero_sources(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
"""No registered sources: the top level is the header line alone —
|
||||||
|
``0 sources:`` (the old ``0 documents:`` behavior preserved in
|
||||||
|
spirit), still a counted result."""
|
||||||
|
monkeypatch.setattr(agent, "ls_top", lambda db: [])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
||||||
[StreamPiece("content", "ans")],
|
[StreamPiece("content", "ans")],
|
||||||
)
|
)
|
||||||
asyncio.run(_run(llm, holder, _settings()))
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
|
assert llm.requests[1][0][3]["content"] == "0 sources:"
|
||||||
assert holder.tool_calls == 1
|
assert holder.tool_calls == 1
|
||||||
|
|
||||||
|
|
||||||
@@ -583,13 +600,12 @@ def test_ls_empty_catalog_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -
|
|||||||
({"path": 7}, "non-string path"),
|
({"path": 7}, "non-string path"),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
def test_ls_blank_path_lists_full_catalog(
|
def test_ls_blank_path_lists_top_level(
|
||||||
monkeypatch: pytest.MonkeyPatch, arguments: dict[str, Any], label: str
|
monkeypatch: pytest.MonkeyPatch, arguments: dict[str, Any], label: str
|
||||||
) -> None:
|
) -> None:
|
||||||
"""A blank (or non-string) ``path`` is treated as omitted — the full
|
"""A blank (or non-string) ``path`` is treated as omitted — the top
|
||||||
catalog, counted (no refusal for an empty scope)."""
|
level (the sources), counted (no refusal for an empty scope)."""
|
||||||
catalog = [("S", "a.md", "A")]
|
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
|
|
||||||
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
@@ -597,21 +613,25 @@ def test_ls_blank_path_lists_full_catalog(
|
|||||||
[StreamPiece("content", "ans")],
|
[StreamPiece("content", "ans")],
|
||||||
)
|
)
|
||||||
asyncio.run(_run(llm, holder, _settings()))
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
assert (
|
assert llm.requests[1][0][3]["content"] == "1 sources:\n\nS — 1 documents"
|
||||||
llm.requests[1][0][3]["content"] == "1 documents:\nsource: S | path: a.md | title: A"
|
|
||||||
)
|
|
||||||
assert holder.tool_calls == 1
|
assert holder.tool_calls == 1
|
||||||
|
|
||||||
|
|
||||||
def test_ls_scoped_to_known_source(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_ls_source_scope_lists_root_folder(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
"""A known source name: the same listing filtered to that source —
|
"""A registered source name (no ``/``): the source's ROOT folder —
|
||||||
|
subfolders (2-space-indented, path order, ``: {summary}`` only when
|
||||||
|
stored) + the root's own file lines in EXACTLY the
|
||||||
|
``source: X | path: Y | title: Z`` format — the pinned template,
|
||||||
counted."""
|
counted."""
|
||||||
catalog = [
|
monkeypatch.setattr(
|
||||||
("Deployments", "backups.md", "Backup Strategy"),
|
agent,
|
||||||
("Homelab", "a.md", "A"),
|
"ls_folder",
|
||||||
("Homelab", "b.md", "B"),
|
lambda db, source, folder: (
|
||||||
]
|
[("backups", 2, "Backup notes."), ("networking", 1, None)],
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
|
[("Homelab", "readme.md", "Readme")],
|
||||||
|
1,
|
||||||
|
),
|
||||||
|
)
|
||||||
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Deployments", "Homelab"])
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Deployments", "Homelab"])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
@@ -620,9 +640,50 @@ def test_ls_scoped_to_known_source(monkeypatch: pytest.MonkeyPatch) -> None:
|
|||||||
)
|
)
|
||||||
asyncio.run(_run(llm, holder, _settings()))
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
assert llm.requests[1][0][3]["content"] == (
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
"2 documents:\n"
|
"Homelab — 1 documents, 2 folders:\n"
|
||||||
"source: Homelab | path: a.md | title: A\n"
|
"\n"
|
||||||
"source: Homelab | path: b.md | title: B"
|
" backups/ — 2 documents: Backup notes.\n"
|
||||||
|
" networking/ — 1 documents\n"
|
||||||
|
"\n"
|
||||||
|
"source: Homelab | path: readme.md | title: Readme"
|
||||||
|
)
|
||||||
|
assert holder.tool_calls == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_nested_folder_scope_lists_one_level_deeper(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
"""A ``source/folder`` path: that folder's subfolders + own file
|
||||||
|
lines, identity = ``source/folder`` (the same template as the
|
||||||
|
root), counted; the fetchers are the source-scoped ones."""
|
||||||
|
|
||||||
|
def _rows(db: Any, source: str) -> list[tuple[str, str]]:
|
||||||
|
assert (source, db) == ("Homelab", None)
|
||||||
|
return [
|
||||||
|
("networking/lan.md", "LAN"),
|
||||||
|
("networking/vpn.md", "VPN"),
|
||||||
|
]
|
||||||
|
|
||||||
|
monkeypatch.setattr(agent, "_source_document_rows", _rows)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
agent, "_source_folder_summaries", lambda db, source: {"networking": "Network notes."}
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
|
||||||
|
holder = AgentHolder()
|
||||||
|
llm = ScriptedLLM(
|
||||||
|
[
|
||||||
|
ToolCallPiece(
|
||||||
|
id="call_1", name="ls", arguments={"path": "Homelab/networking"}
|
||||||
|
)
|
||||||
|
],
|
||||||
|
[StreamPiece("content", "ans")],
|
||||||
|
)
|
||||||
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
|
"Homelab/networking — 2 documents, 0 folders:\n"
|
||||||
|
"\n"
|
||||||
|
"source: Homelab | path: networking/lan.md | title: LAN\n"
|
||||||
|
"source: Homelab | path: networking/vpn.md | title: VPN"
|
||||||
)
|
)
|
||||||
assert holder.tool_calls == 1
|
assert holder.tool_calls == 1
|
||||||
|
|
||||||
@@ -631,9 +692,13 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
|
|||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""A registered source with no indexed documents is KNOWN (the
|
"""A registered source with no indexed documents is KNOWN (the
|
||||||
registry is the source of truth, not the catalog): it lists as
|
registry is the source of truth, not the catalog): it lists its
|
||||||
``0 documents:`` — a valid, counted result, not a refusal."""
|
header line alone (``… — 0 documents, 0 folders:`` — the old
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Other", "a.md", "A")])
|
``0 documents:`` behavior preserved in spirit) — a valid, counted
|
||||||
|
result, not a refusal."""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
agent, "ls_folder", lambda db, source, folder: ([], [], 0)
|
||||||
|
)
|
||||||
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab", "Other"])
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab", "Other"])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
@@ -641,7 +706,7 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
|
|||||||
[StreamPiece("content", "ans")],
|
[StreamPiece("content", "ans")],
|
||||||
)
|
)
|
||||||
asyncio.run(_run(llm, holder, _settings()))
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
|
assert llm.requests[1][0][3]["content"] == "Homelab — 0 documents, 0 folders:"
|
||||||
assert holder.tool_calls == 1 # an executed ls, not a refusal
|
assert holder.tool_calls == 1 # an executed ls, not a refusal
|
||||||
assert llm.requests[1][1] == AGENT_TOOLS
|
assert llm.requests[1][1] == AGENT_TOOLS
|
||||||
|
|
||||||
@@ -650,7 +715,6 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
|
|||||||
"""A ``path`` without ``/`` matching no source name is a refusal —
|
"""A ``path`` without ``/`` matching no source name is a refusal —
|
||||||
the extended line with the teaching parenthetical (phase 72), not
|
the extended line with the teaching parenthetical (phase 72), not
|
||||||
counted, the round cap bounds its repetition."""
|
counted, the round cap bounds its repetition."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
|
||||||
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
@@ -666,20 +730,14 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
|
|||||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||||
|
|
||||||
|
|
||||||
def test_ls_path_like_scope_gets_document_path_teaching_refusal(
|
def test_ls_path_like_scope_unknown_source_gets_no_source_refusal(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Phase 72: a stripped scope containing ``/`` looks like a document
|
"""Phase 94: a ``/`` now names a folder, so the phase-72
|
||||||
path (the incident's ``ls(path='app/rag/importer.py')``) — a source
|
document-path teaching is DELETED — a ``source/…`` argument whose
|
||||||
name is a directory basename and can never contain one, so this gets
|
FIRST segment names no registered source gets the no-source refusal
|
||||||
the ``LS_PATH_NOT_A_SOURCE`` teaching line with the argument echoed;
|
(the segment echoed), counted in nothing, tools stay offered."""
|
||||||
no registry lookup, counts in nothing, tools stay offered."""
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
|
||||||
|
|
||||||
def _boom_sources(*_a: Any, **_k: Any) -> None:
|
|
||||||
raise AssertionError("no registry lookup for a path-like scope")
|
|
||||||
|
|
||||||
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
|
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[
|
[
|
||||||
@@ -695,7 +753,7 @@ def test_ls_path_like_scope_gets_document_path_teaching_refusal(
|
|||||||
assert holder.tool_calls == 0 # a refusal counts in nothing
|
assert holder.tool_calls == 0 # a refusal counts in nothing
|
||||||
assert (
|
assert (
|
||||||
llm.requests[1][0][3]["content"]
|
llm.requests[1][0][3]["content"]
|
||||||
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
|
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
|
||||||
)
|
)
|
||||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||||
|
|
||||||
@@ -706,8 +764,7 @@ def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
|
|||||||
"""Phase 72: ``ls(path='.')`` (the incident's second round — no
|
"""Phase 72: ``ls(path='.')`` (the incident's second round — no
|
||||||
``/``, no matching source) gets the extended no-source refusal with
|
``/``, no matching source) gets the extended no-source refusal with
|
||||||
the teaching parenthetical, ``'.'`` echoed — not counted, tools stay
|
the teaching parenthetical, ``'.'`` echoed — not counted, tools stay
|
||||||
offered."""
|
offered (unchanged by phase 94)."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
|
||||||
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
@@ -723,6 +780,312 @@ def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
|
|||||||
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_unknown_top_level_folder_gets_not_a_folder_teaching(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
"""Phase 94: a ``source/…`` argument whose TOP-LEVEL folder segment
|
||||||
|
matches no indexed prefix gets the NOT-A-FOLDER teaching — the
|
||||||
|
argument echoed, the source named, its direct subfolders listed so
|
||||||
|
the model self-corrects in the next round; not counted, tools stay
|
||||||
|
offered."""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
agent,
|
||||||
|
"_source_document_rows",
|
||||||
|
lambda db, source: [
|
||||||
|
("backups/cron.md", "Cron"),
|
||||||
|
("containers/caddy.md", "Caddy"),
|
||||||
|
("networking/lan.md", "LAN"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
|
||||||
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
|
||||||
|
holder = AgentHolder()
|
||||||
|
llm = ScriptedLLM(
|
||||||
|
[
|
||||||
|
ToolCallPiece(
|
||||||
|
id="call_1", name="ls", arguments={"path": "Homelab/netwoking"}
|
||||||
|
)
|
||||||
|
],
|
||||||
|
[StreamPiece("content", "ans")],
|
||||||
|
)
|
||||||
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
|
assert holder.tool_calls == 0 # a refusal counts in nothing
|
||||||
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
|
"'Homelab/netwoking' is not a folder — Homelab has: "
|
||||||
|
"backups/ containers/ networking/"
|
||||||
|
)
|
||||||
|
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_unknown_nested_folder_gets_not_a_folder_with_nested_parent(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
"""Phase 94: a nested miss names the DEEPEST existing ancestor —
|
||||||
|
``source/folder`` — and lists ITS direct subfolders (bounded: the
|
||||||
|
parent's own listing, no new flood path)."""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
agent,
|
||||||
|
"_source_document_rows",
|
||||||
|
lambda db, source: [
|
||||||
|
("networking/lan/a.md", "A"),
|
||||||
|
("networking/vpn/b.md", "B"),
|
||||||
|
("readme.md", "Readme"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
|
||||||
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
|
||||||
|
holder = AgentHolder()
|
||||||
|
llm = ScriptedLLM(
|
||||||
|
[
|
||||||
|
ToolCallPiece(
|
||||||
|
id="call_1", name="ls", arguments={"path": "Homelab/networking/lan/x"}
|
||||||
|
)
|
||||||
|
],
|
||||||
|
[StreamPiece("content", "ans")],
|
||||||
|
)
|
||||||
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
|
assert holder.tool_calls == 0
|
||||||
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
|
"'Homelab/networking/lan/x' is not a folder — "
|
||||||
|
"Homelab/networking/lan has: none"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_file_path_scope_gets_not_a_folder(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""Phase 94: a document's OWN path is never a folder (nothing starts
|
||||||
|
with ``path + '/'``) — ``ls`` of a file path refuses with the
|
||||||
|
NOT-A-FOLDER teaching (the parent's subfolders listed)."""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
agent,
|
||||||
|
"_source_document_rows",
|
||||||
|
lambda db, source: [("notes.md", "Notes"), ("a/b.md", "B")],
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
|
||||||
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
|
||||||
|
holder = AgentHolder()
|
||||||
|
llm = ScriptedLLM(
|
||||||
|
[
|
||||||
|
ToolCallPiece(id="call_1", name="ls", arguments={"path": "S/notes.md"})
|
||||||
|
],
|
||||||
|
[StreamPiece("content", "ans")],
|
||||||
|
)
|
||||||
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
|
assert holder.tool_calls == 0
|
||||||
|
assert llm.requests[1][0][3]["content"] == (
|
||||||
|
"'S/notes.md' is not a folder — S has: a/"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- ls_top / ls_folder: the drill-down accessors (pure + composed) ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_top_registry_order_zero_docs_and_summaries(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
"""``ls_top``: registry order (not catalog order), a 0-document
|
||||||
|
source still lists, the summary is the stored ``(source, "")`` row
|
||||||
|
or ``None`` when absent; an empty registry → ``[]``."""
|
||||||
|
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Zeta", "Alpha"])
|
||||||
|
monkeypatch.setattr(
|
||||||
|
agent, "_source_document_counts", lambda db: [("Zeta", 3), ("Beta", 1)]
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
agent, "_source_root_summaries", lambda db: [("Zeta", "Zeta stuff.")]
|
||||||
|
)
|
||||||
|
assert agent.ls_top(cast("Session", object())) == [
|
||||||
|
("Zeta", 3, "Zeta stuff."),
|
||||||
|
("Alpha", 0, None), # 0 docs (no count row) + no stored summary
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_top_empty_registry_is_empty(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
monkeypatch.setattr(agent, "list_source_names", lambda db: [])
|
||||||
|
|
||||||
|
def _boom(*_a: Any, **_k: Any) -> None:
|
||||||
|
raise AssertionError("no fetches for an empty registry")
|
||||||
|
|
||||||
|
monkeypatch.setattr(agent, "_source_document_counts", _boom)
|
||||||
|
monkeypatch.setattr(agent, "_source_root_summaries", _boom)
|
||||||
|
assert agent.ls_top(cast("Session", object())) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_ls_folder_composes_the_fetchers(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""``ls_folder`` = the source's document rows + stored summaries
|
||||||
|
through the pure :func:`group_folder_listing` (the fetchers are the
|
||||||
|
monkeypatch surface)."""
|
||||||
|
seen: list[tuple[str, str, str]] = []
|
||||||
|
|
||||||
|
def _rows(db: Any, source: str) -> list[tuple[str, str]]:
|
||||||
|
seen.append(("rows", source, ""))
|
||||||
|
return [("a/b.md", "B"), ("a.md", "A")]
|
||||||
|
|
||||||
|
def _summaries(db: Any, source: str) -> dict[str, str]:
|
||||||
|
seen.append(("summaries", source, ""))
|
||||||
|
return {"a": "A stuff."}
|
||||||
|
|
||||||
|
monkeypatch.setattr(agent, "_source_document_rows", _rows)
|
||||||
|
monkeypatch.setattr(agent, "_source_folder_summaries", _summaries)
|
||||||
|
assert agent.ls_folder(cast("Session", object()), "S", "") == (
|
||||||
|
[("a", 1, "A stuff.")],
|
||||||
|
[("S", "a.md", "A")],
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
assert seen == [("rows", "S", ""), ("summaries", "S", "")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_folder_listing_subfolder_recursion_and_counts() -> None:
|
||||||
|
"""The recursive count per subfolder — every path equal to the
|
||||||
|
folder or starting with ``folder + '/'`` (a doc under ``a/b/``
|
||||||
|
counts for BOTH ``a`` and ``a/b``), path order, the stored summary
|
||||||
|
attached or ``None``."""
|
||||||
|
rows = [
|
||||||
|
("a/b/c.md", "C"),
|
||||||
|
("a/b/d.md", "D"),
|
||||||
|
("a/e.md", "E"),
|
||||||
|
("f.md", "F"),
|
||||||
|
]
|
||||||
|
sub, files, total = agent.group_folder_listing(
|
||||||
|
"S", "", rows, {"a": "A subtree.", "a/b": "B subtree."}
|
||||||
|
)
|
||||||
|
# ROOT level: the direct subfolders of "" are the TOP-LEVEL folders
|
||||||
|
# only (a/b is nested under a, not direct) — a's count is its whole
|
||||||
|
# recursive subtree (a/e.md + a/b/c.md + a/b/d.md), the stored
|
||||||
|
# summary attached.
|
||||||
|
assert sub == [("a", 3, "A subtree.")]
|
||||||
|
assert files == [("S", "f.md", "F")]
|
||||||
|
assert total == 1
|
||||||
|
# One level down: a/b is a's direct subfolder with its own count.
|
||||||
|
sub2, _files2, _total2 = agent.group_folder_listing("S", "a", rows, {"a/b": "B subtree."})
|
||||||
|
assert sub2 == [("a/b", 2, "B subtree.")]
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_folder_listing_nested_level_counts_and_membership() -> None:
|
||||||
|
"""One level down: ``a``'s direct subfolder is ``a/b`` (count 2),
|
||||||
|
its own direct file is ``a/e.md`` (``a/b/c.md`` is NOT a direct
|
||||||
|
file of ``a``) — membership is the folder_of rule, order is path
|
||||||
|
order."""
|
||||||
|
rows = [
|
||||||
|
("a/b/c.md", "C"),
|
||||||
|
("a/b/d.md", "D"),
|
||||||
|
("a/e.md", "E"),
|
||||||
|
]
|
||||||
|
sub, files, total = agent.group_folder_listing("S", "a", rows, {})
|
||||||
|
assert sub == [("a/b", 2, None)]
|
||||||
|
assert files == [("S", "a/e.md", "E")]
|
||||||
|
assert total == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_folder_listing_file_path_is_not_a_folder() -> None:
|
||||||
|
"""A document whose path is a prefix of NO other path is a file,
|
||||||
|
never a folder: ``ls`` of it must not list a subfolder (and the
|
||||||
|
``path == folder`` count arm only fires for TRUE folders — a doc
|
||||||
|
sharing a real folder's name counts for that folder, the existence
|
||||||
|
rule intact)."""
|
||||||
|
rows = [
|
||||||
|
("a.md", "A"), # a file at the root, and a folder name? NO —
|
||||||
|
("b/x.md", "X"), # nothing starts with "a.md/"
|
||||||
|
]
|
||||||
|
sub, files, total = agent.group_folder_listing("S", "", rows, {})
|
||||||
|
assert sub == [("b", 1, None)] # "a.md" is NOT a subfolder
|
||||||
|
assert files == [("S", "a.md", "A")] # b/x.md is NOT a direct root file
|
||||||
|
assert total == 1
|
||||||
|
# The path == folder arm: a doc named "a" under a real folder "a/".
|
||||||
|
rows2 = [("a", "FileA"), ("a/c.md", "C")]
|
||||||
|
sub2, files2, total2 = agent.group_folder_listing("S", "", rows2, {})
|
||||||
|
assert sub2 == [("a", 2, None)] # the file "a" counts for folder "a"
|
||||||
|
assert files2 == [("S", "a", "FileA")] # …and is a direct ROOT file
|
||||||
|
assert total2 == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_folder_listing_caps_files_at_fifty_keeps_the_total() -> None:
|
||||||
|
"""The cap: 51 direct files → 50 file lines + the PRE-cap total (51)
|
||||||
|
for the renderer's note; 50 files → 50 lines, no note material.
|
||||||
|
A 500-file folder costs 50 lines, never 500."""
|
||||||
|
rows51 = [(f"big/f{i:03d}.md", f"T{i}") for i in range(51)]
|
||||||
|
sub, files, total = agent.group_folder_listing("S", "big", rows51, {})
|
||||||
|
assert sub == []
|
||||||
|
assert total == 51
|
||||||
|
assert len(files) == 50
|
||||||
|
assert files[0] == ("S", "big/f000.md", "T0")
|
||||||
|
assert files[-1] == ("S", "big/f049.md", "T49")
|
||||||
|
rows50 = [(f"big/f{i:03d}.md", f"T{i}") for i in range(50)]
|
||||||
|
_sub, files50, total50 = agent.group_folder_listing("S", "big", rows50, {})
|
||||||
|
assert total50 == 50 and len(files50) == 50
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- the pinned drill-down templates (byte-for-byte) ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_ls_top_template() -> None:
|
||||||
|
assert (
|
||||||
|
agent.render_ls_top(
|
||||||
|
[("Deployments", 3, None), ("Homelab", 5, "Home lab notes.")]
|
||||||
|
)
|
||||||
|
== "2 sources:\n\n"
|
||||||
|
"Deployments — 3 documents\n"
|
||||||
|
"Homelab — 5 documents\n"
|
||||||
|
" Home lab notes."
|
||||||
|
)
|
||||||
|
assert agent.render_ls_top([]) == "0 sources:"
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_folder_listing_root_template() -> None:
|
||||||
|
assert (
|
||||||
|
agent.render_folder_listing(
|
||||||
|
"Homelab",
|
||||||
|
[("backups", 2, "Backup notes."), ("networking", 1, None)],
|
||||||
|
[("Homelab", "readme.md", "Readme")],
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
== "Homelab — 1 documents, 2 folders:\n"
|
||||||
|
"\n"
|
||||||
|
" backups/ — 2 documents: Backup notes.\n"
|
||||||
|
" networking/ — 1 documents\n"
|
||||||
|
"\n"
|
||||||
|
"source: Homelab | path: readme.md | title: Readme"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_folder_listing_empty_level_is_header_alone() -> None:
|
||||||
|
"""A registered source with no documents: the header line alone —
|
||||||
|
the old ``0 documents:`` behavior preserved in spirit."""
|
||||||
|
assert agent.render_folder_listing("Homelab", [], [], 0) == (
|
||||||
|
"Homelab — 0 documents, 0 folders:"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_folder_listing_subfolders_only_no_blank_trailer() -> None:
|
||||||
|
"""Subfolders but no own files: header + blank + subfolder lines —
|
||||||
|
no trailing blank line, no file section."""
|
||||||
|
assert (
|
||||||
|
agent.render_folder_listing("S", [("a", 1, None)], [], 0)
|
||||||
|
== "S — 0 documents, 1 folders:\n\n a/ — 1 documents"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_folder_listing_cap_note_only_past_fifty() -> None:
|
||||||
|
"""The note appears ONLY when the folder's own files outnumber the
|
||||||
|
cap: 51 → 50 lines + the deterministic grep-pointer note (the
|
||||||
|
``…and 1 more…`` shape — unpluralized, the house pin); 50 → no
|
||||||
|
note."""
|
||||||
|
files51 = [("S", f"f{i:03d}.md", f"T{i}") for i in range(51)]
|
||||||
|
capped = files51[:50]
|
||||||
|
rendered = agent.render_folder_listing("S/big", [], capped, 51)
|
||||||
|
lines = rendered.splitlines()
|
||||||
|
assert lines[0] == "S/big — 51 documents, 0 folders:"
|
||||||
|
assert len(lines) == 1 + 1 + 50 + 1 # header, blank, 50 lines, note
|
||||||
|
assert lines[-1] == (
|
||||||
|
"…and 1 more documents in this folder — use grep (pattern) to "
|
||||||
|
"find a specific one."
|
||||||
|
)
|
||||||
|
files50 = [("S", f"f{i:03d}.md", f"T{i}") for i in range(50)]
|
||||||
|
rendered50 = agent.render_folder_listing("S/big", [], files50, 50)
|
||||||
|
assert rendered50.splitlines()[-1] == "source: S | path: f049.md | title: T49"
|
||||||
|
assert "more documents" not in rendered50
|
||||||
|
|
||||||
|
|
||||||
# ---------- read: the canonical combined source/path form ----------
|
# ---------- read: the canonical combined source/path form ----------
|
||||||
|
|
||||||
|
|
||||||
@@ -771,7 +1134,6 @@ def test_read_bare_source_name_refused_without_db(monkeypatch: pytest.MonkeyPatc
|
|||||||
no-document refusal (the argument echoed as passed), no DB lookup
|
no-document refusal (the argument echoed as passed), no DB lookup
|
||||||
(NOT even the phase-72 candidate lookup — ``all_documents`` must
|
(NOT even the phase-72 candidate lookup — ``all_documents`` must
|
||||||
not run either), nothing counted."""
|
not run either), nothing counted."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Homelab", "a.md", "A")])
|
|
||||||
|
|
||||||
def _boom(*_a: Any, **_k: Any) -> None:
|
def _boom(*_a: Any, **_k: Any) -> None:
|
||||||
raise AssertionError(
|
raise AssertionError(
|
||||||
@@ -1158,7 +1520,6 @@ def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch)
|
|||||||
|
|
||||||
|
|
||||||
def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[ToolCallPiece(id="call_1", name="delete_universe", arguments={"x": 1})],
|
[ToolCallPiece(id="call_1", name="delete_universe", arguments={"x": 1})],
|
||||||
@@ -1702,13 +2063,11 @@ def test_grep_counts_but_never_adds_context(monkeypatch: pytest.MonkeyPatch) ->
|
|||||||
|
|
||||||
|
|
||||||
def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
"""Re-lists execute — a second ``ls`` in one turn returns the catalog
|
"""Re-lists execute — a second ``ls`` in one turn returns the top
|
||||||
again and counts in ``tool_calls`` (no budget to exhaust)."""
|
level again and counts in ``tool_calls`` (no budget to exhaust)."""
|
||||||
catalog = [
|
monkeypatch.setattr(
|
||||||
("Deployments", "backups.md", "Backup Strategy"),
|
agent, "ls_top", lambda db: [("Deployments", 1, None), ("Homelab", 1, None)]
|
||||||
("Homelab", "aws-route53.md", "AWS Route53 Records"),
|
)
|
||||||
]
|
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
|
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
||||||
@@ -1717,12 +2076,8 @@ def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
|
|||||||
)
|
)
|
||||||
asyncio.run(_run(llm, holder, _settings()))
|
asyncio.run(_run(llm, holder, _settings()))
|
||||||
assert holder.tool_calls == 2 # both re-lists executed and counted
|
assert holder.tool_calls == 2 # both re-lists executed and counted
|
||||||
listing = (
|
listing = "2 sources:\n\nDeployments — 1 documents\nHomelab — 1 documents"
|
||||||
"2 documents:\n"
|
# The answer request carries the listing a second time as a tool result.
|
||||||
"source: Deployments | path: backups.md | title: Backup Strategy\n"
|
|
||||||
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
|
|
||||||
)
|
|
||||||
# The answer request carries the catalog a second time as a tool result.
|
|
||||||
assert llm.requests[2][0][3]["content"] == listing # first listing
|
assert llm.requests[2][0][3]["content"] == listing # first listing
|
||||||
assert llm.requests[2][0][5]["content"] == listing # the re-list
|
assert llm.requests[2][0][5]["content"] == listing # the re-list
|
||||||
assert llm.requests[2][1] == AGENT_TOOLS # still offered (no budgets)
|
assert llm.requests[2][1] == AGENT_TOOLS # still offered (no budgets)
|
||||||
@@ -1758,8 +2113,8 @@ def test_always_ls_bounded_by_round_cap(monkeypatch: pytest.MonkeyPatch) -> None
|
|||||||
"""A model that keeps calling ``ls`` gets exactly
|
"""A model that keeps calling ``ls`` gets exactly
|
||||||
``agent_max_rounds`` tool rounds, then one forced ``tools=None``
|
``agent_max_rounds`` tool rounds, then one forced ``tools=None``
|
||||||
request streams the answer — the cap is the only forced exit."""
|
request streams the answer — the cap is the only forced exit."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
|
||||||
listing = "1 documents:\nsource: S | path: a.md | title: A"
|
listing = "1 sources:\n\nS — 1 documents"
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
||||||
@@ -1923,7 +2278,7 @@ def test_round_retried_before_first_piece(
|
|||||||
same messages: the stream carries a RetryPiece BEFORE the tool call,
|
same messages: the stream carries a RetryPiece BEFORE the tool call,
|
||||||
the tool executes, the final answer streams, and the per-call log line
|
the tool executes, the final answer streams, and the per-call log line
|
||||||
is still emitted exactly once (retries are invisible to the loop)."""
|
is still emitted exactly once (retries are invisible to the loop)."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = FailingLLM(
|
llm = FailingLLM(
|
||||||
[
|
[
|
||||||
@@ -1961,7 +2316,6 @@ def test_round_failure_after_first_piece_is_terminal(monkeypatch: pytest.MonkeyP
|
|||||||
the LLMError propagates out of ``run_agent``, no RetryPiece, no
|
the LLMError propagates out of ``run_agent``, no RetryPiece, no
|
||||||
sleep, no second request, and the holder is untouched (the tool
|
sleep, no second request, and the holder is untouched (the tool
|
||||||
never ran)."""
|
never ran)."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = FailingLLM(
|
llm = FailingLLM(
|
||||||
[([StreamPiece("content", "partial ")], LLMError("mid-stream drop"))]
|
[([StreamPiece("content", "partial ")], LLMError("mid-stream drop"))]
|
||||||
@@ -1994,7 +2348,7 @@ def test_forced_final_no_tools_call_is_retried(monkeypatch: pytest.MonkeyPatch)
|
|||||||
"""The forced final request (round cap reached) goes through the same
|
"""The forced final request (round cap reached) goes through the same
|
||||||
retry rule: a failure before its first piece yields a RetryPiece and
|
retry rule: a failure before its first piece yields a RetryPiece and
|
||||||
restarts with ``tools=None``; the answer from the retry streams."""
|
restarts with ``tools=None``; the answer from the retry streams."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = FailingLLM(
|
llm = FailingLLM(
|
||||||
[
|
[
|
||||||
@@ -2026,7 +2380,6 @@ def test_zero_retries_is_one_plain_attempt(monkeypatch: pytest.MonkeyPatch) -> N
|
|||||||
"""The kill-switch path (``llm_retries=0``): a dead round raises
|
"""The kill-switch path (``llm_retries=0``): a dead round raises
|
||||||
immediately — one request, no RetryPiece, no sleep (pre-phase-67
|
immediately — one request, no RetryPiece, no sleep (pre-phase-67
|
||||||
behavior)."""
|
behavior)."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = FailingLLM([([], LLMError("connection refused"))])
|
llm = FailingLLM([([], LLMError("connection refused"))])
|
||||||
sleeps = _record_sleeps(monkeypatch)
|
sleeps = _record_sleeps(monkeypatch)
|
||||||
@@ -2065,7 +2418,6 @@ def test_abandon_mid_retry_sleep_leaks_nothing(monkeypatch: pytest.MonkeyPatch)
|
|||||||
await asyncio.Event().wait() # park until the abandon arrives
|
await asyncio.Event().wait() # park until the abandon arrives
|
||||||
|
|
||||||
monkeypatch.setattr(asyncio, "sleep", parking_sleep)
|
monkeypatch.setattr(asyncio, "sleep", parking_sleep)
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
|
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = FailingLLM(
|
llm = FailingLLM(
|
||||||
[([], LLMError("endpoint down")), ([StreamPiece("content", "never")], None)]
|
[([], LLMError("endpoint down")), ([StreamPiece("content", "never")], None)]
|
||||||
@@ -2111,7 +2463,7 @@ def test_retries_are_invisible_to_the_round_cap(
|
|||||||
2, the retried first round and the second tool round fill the cap —
|
2, the retried first round and the second tool round fill the cap —
|
||||||
the forced final follows the SECOND call, and the log lines read
|
the forced final follows the SECOND call, and the log lines read
|
||||||
round=1/2 and round=2/2."""
|
round=1/2 and round=2/2."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = FailingLLM(
|
llm = FailingLLM(
|
||||||
[
|
[
|
||||||
@@ -2364,7 +2716,7 @@ def test_scaffolding_round_with_tool_calls_needs_no_recovery(
|
|||||||
ran, and the policy keys on the no-calls exit only — no recovery (the
|
ran, and the policy keys on the no-calls exit only — no recovery (the
|
||||||
next round is a normal tools-offered round carrying the tool
|
next round is a normal tools-offered round carrying the tool
|
||||||
history)."""
|
history)."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[
|
[
|
||||||
@@ -2398,7 +2750,7 @@ def test_recovery_after_tool_rounds_keeps_the_history(
|
|||||||
keeps the SINGLE (folded) system message at the front and the tool
|
keeps the SINGLE (folded) system message at the front and the tool
|
||||||
history intact behind it — no second system message, no duplicated
|
history intact behind it — no second system message, no duplicated
|
||||||
correction."""
|
correction."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
||||||
@@ -2423,7 +2775,7 @@ def test_recovery_after_tool_rounds_keeps_the_history(
|
|||||||
assert recovered[3] == {
|
assert recovered[3] == {
|
||||||
"role": "tool",
|
"role": "tool",
|
||||||
"tool_call_id": "call_1",
|
"tool_call_id": "call_1",
|
||||||
"content": "1 documents:\nsource: S | path: a.md | title: A",
|
"content": "1 sources:\n\nS — 1 documents",
|
||||||
}
|
}
|
||||||
assert sum(1 for m in recovered if m["role"] == "system") == 1
|
assert sum(1 for m in recovered if m["role"] == "system") == 1
|
||||||
assert holder.scaffold_stripped == len(_INCIDENT_SPAN)
|
assert holder.scaffold_stripped == len(_INCIDENT_SPAN)
|
||||||
@@ -2436,7 +2788,7 @@ def test_forced_final_scaffolding_only_settles_malformed(
|
|||||||
scaffolding-only forced answer never reaches the user raw — the turn
|
scaffolding-only forced answer never reaches the user raw — the turn
|
||||||
settles with :class:`MalformedReplyError` (the same terminal
|
settles with :class:`MalformedReplyError` (the same terminal
|
||||||
semantics; this turn used no recovery, so nothing is doubled up)."""
|
semantics; this turn used no recovery, so nothing is doubled up)."""
|
||||||
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
|
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
|
||||||
holder = AgentHolder()
|
holder = AgentHolder()
|
||||||
llm = ScriptedLLM(
|
llm = ScriptedLLM(
|
||||||
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
[ToolCallPiece(id="call_1", name="ls", arguments={})],
|
||||||
|
|||||||
@@ -0,0 +1,628 @@
|
|||||||
|
"""Unit: folder summary storage + generator (phase 94, task 01).
|
||||||
|
|
||||||
|
The prompt/grouping tests are pure (no DB): ``FOLDER_SUMMARY_MODE``
|
||||||
|
system prompt, the ``folder_of`` / ``group_by_folder`` recursive-subtree
|
||||||
|
concept, and the user-message cap with the shared ``[…truncated…]``
|
||||||
|
marker. The generator tests run against the local compose Postgres
|
||||||
|
(preferred — real upsert/prune on the ``folder_summaries`` table),
|
||||||
|
skipping with clear instructions when the stack is not up — same
|
||||||
|
pattern as ``test_overview.py``.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import uuid
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import text
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from app.config import Settings, get_settings
|
||||||
|
from app.db import SessionLocal
|
||||||
|
from app.models import Document, FolderSummary
|
||||||
|
from app.rag.folder_summaries import (
|
||||||
|
FOLDER_HEADER_PREFIX,
|
||||||
|
FOLDER_SUMMARY_INSTRUCTION,
|
||||||
|
FOLDER_SUMMARY_MODE,
|
||||||
|
MIN_DOCS_PER_FOLDER,
|
||||||
|
SYSTEM_PROMPT,
|
||||||
|
build_folder_summary_prompt,
|
||||||
|
folder_of,
|
||||||
|
folder_summary_table_empty,
|
||||||
|
generate_folder_summaries,
|
||||||
|
group_by_folder,
|
||||||
|
summarize_folder,
|
||||||
|
)
|
||||||
|
from app.rag.llm import LLMError
|
||||||
|
from app.rag.retriever import TRUNCATION_MARKER
|
||||||
|
from tests.e2e.mock_llm import compose_answer
|
||||||
|
|
||||||
|
REPLY = "Covers lab automation runbooks: inventories, playbooks, and schedules."
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeLLM:
|
||||||
|
"""Duck-typed stand-in for ``LLMClient`` (``chat`` + ``settings``).
|
||||||
|
|
||||||
|
Records each ``(system, user)`` request and the ``model`` kwarg;
|
||||||
|
returns the canned reply, or raises — either a fixed exception or a
|
||||||
|
per-folder failure keyed on the user message's ``Folder: …`` header
|
||||||
|
(the per-folder fail-soft tests).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
reply: str = REPLY,
|
||||||
|
fail_folders: tuple[str, ...] = (),
|
||||||
|
fail: Exception | None = None,
|
||||||
|
) -> None:
|
||||||
|
self._reply = reply
|
||||||
|
self._fail_folders = tuple(fail_folders)
|
||||||
|
self._fail = fail
|
||||||
|
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||||
|
self.calls = 0
|
||||||
|
self.model: str | None = None
|
||||||
|
self.requests: list[tuple[str, str]] = []
|
||||||
|
|
||||||
|
async def chat(
|
||||||
|
self, messages: list[dict[str, str]], model: str | None = None
|
||||||
|
) -> str:
|
||||||
|
self.calls += 1
|
||||||
|
self.model = model
|
||||||
|
system = messages[0]["content"]
|
||||||
|
user = messages[-1]["content"]
|
||||||
|
self.requests.append((system, user))
|
||||||
|
for folder in self._fail_folders:
|
||||||
|
if FOLDER_HEADER_PREFIX + folder in user:
|
||||||
|
raise LLMError(f"simulated lite-model failure for {folder}")
|
||||||
|
if self._fail is not None:
|
||||||
|
raise self._fail
|
||||||
|
return self._reply
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- folder_of ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_folder_of_root_level_file_is_empty() -> None:
|
||||||
|
assert folder_of("a.md") == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_folder_of_one_level() -> None:
|
||||||
|
assert folder_of("a/b.md") == "a"
|
||||||
|
|
||||||
|
|
||||||
|
def test_folder_of_deep_path() -> None:
|
||||||
|
assert folder_of("a/b/c/d.md") == "a/b/c"
|
||||||
|
|
||||||
|
|
||||||
|
def test_folder_of_iterating_walks_prefixes_to_root() -> None:
|
||||||
|
"""Iterating ``folder_of`` over its own result walks the folder
|
||||||
|
prefixes nearest-first, ending at the root (the grouping walk)."""
|
||||||
|
folder = folder_of("a/b/c.md")
|
||||||
|
chain: list[str] = []
|
||||||
|
while folder:
|
||||||
|
chain.append(folder)
|
||||||
|
folder = folder_of(folder)
|
||||||
|
assert chain == ["a/b", "a"] # plus the "" root the grouping adds
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- group_by_folder ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_by_folder_root_file_lands_only_in_source_root() -> None:
|
||||||
|
rows = [("S", "top.md", "T", None)]
|
||||||
|
groups = group_by_folder(rows)
|
||||||
|
assert set(groups) == {("S", "")}
|
||||||
|
assert groups[("S", "")] == rows
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_by_folder_nested_multi_source_recursive_subtree() -> None:
|
||||||
|
"""A doc under ``a/b/`` is present in the ``a``, ``a/b``, and ``""``
|
||||||
|
groups (recursive subtree — the ``ls`` count scope, one concept);
|
||||||
|
per source the candidates are ``""`` + every distinct folder
|
||||||
|
prefix; group lists keep the input (catalogue) order."""
|
||||||
|
rows = [
|
||||||
|
("S", "a/b/c.md", "C", None),
|
||||||
|
("S", "a/b/d.md", "D", None),
|
||||||
|
("S", "a/x.md", "X", None),
|
||||||
|
("S", "top.md", "T", None),
|
||||||
|
("T", "a/b/e.md", "E", None),
|
||||||
|
]
|
||||||
|
groups = group_by_folder(rows)
|
||||||
|
assert set(groups) == {
|
||||||
|
("S", ""),
|
||||||
|
("S", "a"),
|
||||||
|
("S", "a/b"),
|
||||||
|
("T", ""),
|
||||||
|
("T", "a"),
|
||||||
|
("T", "a/b"),
|
||||||
|
}
|
||||||
|
# The recursive-subtree concept: a/b/ docs in the a/, a/b/, and ""
|
||||||
|
# groups alike — exactly the set each level's ls count shows.
|
||||||
|
assert [r[1] for r in groups[("S", "a/b")]] == ["a/b/c.md", "a/b/d.md"]
|
||||||
|
assert [r[1] for r in groups[("S", "a")]] == ["a/b/c.md", "a/b/d.md", "a/x.md"]
|
||||||
|
assert [r[1] for r in groups[("S", "")]] == [
|
||||||
|
"a/b/c.md",
|
||||||
|
"a/b/d.md",
|
||||||
|
"a/x.md",
|
||||||
|
"top.md",
|
||||||
|
]
|
||||||
|
# Multi-source: the same folder prefix under another source is a
|
||||||
|
# separate group (PK is (source, folder_path)).
|
||||||
|
assert [r[1] for r in groups[("T", "a/b")]] == ["a/b/e.md"]
|
||||||
|
assert [r[1] for r in groups[("T", "")]] == ["a/b/e.md"]
|
||||||
|
# Input (catalogue) order is preserved inside each group.
|
||||||
|
assert [r[2] for r in groups[("S", "")]] == ["C", "D", "X", "T"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_by_folder_single_doc_folder_is_a_group_too() -> None:
|
||||||
|
"""Grouping is pure subtree membership (≥ 1 docs): the ≥ 2 rule is
|
||||||
|
the GENERATOR's (the recursive count below the minimum yields no
|
||||||
|
row — pinned by the generator tests, not the grouping)."""
|
||||||
|
rows = [("S", "a/only.md", "O", None)]
|
||||||
|
groups = group_by_folder(rows)
|
||||||
|
assert len(groups[("S", "a")]) == 1 # present, but below the minimum
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_by_folder_doc_path_equal_to_a_folder_prefix_counts_for_it() -> None:
|
||||||
|
"""The count rule's ``path == folder`` arm: a document whose path
|
||||||
|
IS one of the source's folder prefixes (a file sharing its name
|
||||||
|
with a directory) belongs to that folder's group too — the grouping
|
||||||
|
stays EXACTLY the set the ``ls`` count rule counts (path equal or
|
||||||
|
starting with ``folder + "/"``), while the returned keys remain the
|
||||||
|
true folder prefixes only (no file-path keys)."""
|
||||||
|
rows = [
|
||||||
|
("S", "a/b", "B", None), # a file named "b" ... (its path is a folder prefix)
|
||||||
|
("S", "a/b/c.md", "C", None), # ... and a real folder "a/b/" holding a doc
|
||||||
|
("S", "a/x.md", "X", None),
|
||||||
|
]
|
||||||
|
groups = group_by_folder(rows)
|
||||||
|
assert set(groups) == {("S", ""), ("S", "a"), ("S", "a/b")}, (
|
||||||
|
"the keys stay the true folder prefixes — the file's own path adds no key"
|
||||||
|
)
|
||||||
|
assert [r[1] for r in groups[("S", "a/b")]] == ["a/b", "a/b/c.md"]
|
||||||
|
assert [r[1] for r in groups[("S", "a")]] == ["a/b", "a/b/c.md", "a/x.md"]
|
||||||
|
assert [r[1] for r in groups[("S", "")]] == ["a/b", "a/b/c.md", "a/x.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_by_folder_plain_file_path_is_not_a_group_key() -> None:
|
||||||
|
"""A file path that is NO folder prefix (no doc under it) adds no
|
||||||
|
group key of its own — the ``path == folder`` arm only fires when
|
||||||
|
the path really is a prefix of the catalogue."""
|
||||||
|
rows = [("S", "top.md", "T", None), ("S", "a/one.md", "O", None)]
|
||||||
|
groups = group_by_folder(rows)
|
||||||
|
assert set(groups) == {("S", ""), ("S", "a")}
|
||||||
|
assert "top.md" not in [folder for _source, folder in groups]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- build_folder_summary_prompt: system ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_system_prompt_has_marker_and_locked_instruction() -> None:
|
||||||
|
assert SYSTEM_PROMPT.startswith(FOLDER_SUMMARY_MODE)
|
||||||
|
assert FOLDER_SUMMARY_INSTRUCTION in SYSTEM_PROMPT
|
||||||
|
for fragment in (
|
||||||
|
"1-3 sentence",
|
||||||
|
"plain-text summary",
|
||||||
|
"natural language",
|
||||||
|
"Do not use markdown",
|
||||||
|
"not in the list",
|
||||||
|
):
|
||||||
|
assert fragment in SYSTEM_PROMPT
|
||||||
|
system, _ = build_folder_summary_prompt("S", "a/b", [])
|
||||||
|
assert system == SYSTEM_PROMPT
|
||||||
|
assert FOLDER_SUMMARY_MODE in system # the marker the E2E mock keys on
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- build_folder_summary_prompt: user ----------
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_prompt_header_names_the_folder() -> None:
|
||||||
|
"""The first line is the ``FOLDER_HEADER_PREFIX`` header the E2E
|
||||||
|
mock parses: ``<source>`` for the root, ``<source>/<folder_path>``
|
||||||
|
for a folder."""
|
||||||
|
_, user = build_folder_summary_prompt("Homelab", "deployments/ansible", [])
|
||||||
|
assert user == FOLDER_HEADER_PREFIX + "Homelab/deployments/ansible"
|
||||||
|
_, user = build_folder_summary_prompt("Homelab", "", [])
|
||||||
|
assert user == FOLDER_HEADER_PREFIX + "Homelab"
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_lines_carry_path_title_and_first_summary_line() -> None:
|
||||||
|
docs = [
|
||||||
|
("S", "a/b/one.md", "One", "First lead.\nSecond line.\nSource: S/a/b/one.md"),
|
||||||
|
("S", "a/b/two.md", "Two", None),
|
||||||
|
]
|
||||||
|
system, user = build_folder_summary_prompt("S", "a/b", docs)
|
||||||
|
assert system == SYSTEM_PROMPT
|
||||||
|
assert user == (
|
||||||
|
"Folder: S/a/b\n"
|
||||||
|
"a/b/one.md — One — First lead.\n"
|
||||||
|
"a/b/two.md — Two"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_line_omits_summary_field_when_absent_or_blank() -> None:
|
||||||
|
docs = [
|
||||||
|
("S", "a/x.md", "X", None),
|
||||||
|
("S", "a/y.md", "Y", " \n\t "),
|
||||||
|
]
|
||||||
|
_, user = build_folder_summary_prompt("S", "a", docs)
|
||||||
|
assert user == "Folder: S/a\na/x.md — X\na/y.md — Y"
|
||||||
|
assert " — " in user # the path — title join only
|
||||||
|
assert not any(line.endswith(" — ") for line in user.splitlines())
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_line_uses_only_first_summary_line() -> None:
|
||||||
|
docs = [
|
||||||
|
("S", "a/x.md", "X", "First line.\nSecond line.\nSource: S/a/x.md"),
|
||||||
|
]
|
||||||
|
_, user = build_folder_summary_prompt("S", "a", docs)
|
||||||
|
assert user == "Folder: S/a\na/x.md — X — First line."
|
||||||
|
assert "Second line" not in user
|
||||||
|
assert "Source:" not in user
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_prompt_truncated_with_marker_when_over_custom_cap() -> None:
|
||||||
|
docs = [("S", f"a/f{i}.md", f"T{i}", None) for i in range(10)]
|
||||||
|
_, full = build_folder_summary_prompt("S", "a", docs, max_chars=10_000)
|
||||||
|
cap = 30
|
||||||
|
_, user = build_folder_summary_prompt("S", "a", docs, max_chars=cap)
|
||||||
|
assert user == full[:cap] + "\n" + TRUNCATION_MARKER
|
||||||
|
assert user.endswith(TRUNCATION_MARKER)
|
||||||
|
assert len(user) > cap # the marker makes the cut visible past the cap
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_prompt_at_exact_cap_not_truncated() -> None:
|
||||||
|
docs = [("S", "a/x.md", "X", None)] # "Folder: S/a\na/x.md — X" = 22 chars
|
||||||
|
_, user = build_folder_summary_prompt("S", "a", docs, max_chars=22)
|
||||||
|
assert user == "Folder: S/a\na/x.md — X"
|
||||||
|
assert TRUNCATION_MARKER not in user
|
||||||
|
|
||||||
|
|
||||||
|
def test_user_prompt_truncated_at_default_cap() -> None:
|
||||||
|
"""No explicit cap → ``BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS`` (read
|
||||||
|
from the live settings, so the test holds for any configured
|
||||||
|
value)."""
|
||||||
|
cap = get_settings().folder_summary_input_max_chars
|
||||||
|
docs = [("S", f"a/f{i}.md", "T", None) for i in range(3_000)]
|
||||||
|
_, user = build_folder_summary_prompt("S", "a", docs)
|
||||||
|
assert user.endswith(TRUNCATION_MARKER)
|
||||||
|
body = user.removesuffix("\n" + TRUNCATION_MARKER)
|
||||||
|
assert len(body) == cap # cut exactly at the cap, marker on its own line
|
||||||
|
assert "f2999.md" not in body # the overflow never reaches the model
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- summarize_folder ----------
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- the E2E mock's FOLDER_SUMMARY_MODE branch ----------
|
||||||
|
|
||||||
|
|
||||||
|
def _mock_body(system: str, user: str) -> dict[str, Any]:
|
||||||
|
"""A minimal chat-completion body for the mock's ``compose_answer``."""
|
||||||
|
return {"messages": [
|
||||||
|
{"role": "system", "content": system},
|
||||||
|
{"role": "user", "content": user},
|
||||||
|
]}
|
||||||
|
|
||||||
|
|
||||||
|
def test_mock_returns_canned_folder_summary_naming_the_folder() -> None:
|
||||||
|
"""The deterministic E2E mock keys on the ``FOLDER_SUMMARY_MODE``
|
||||||
|
marker in the system prompt and returns the canned one-liner naming
|
||||||
|
the folder from the ``Folder: …`` header — driven through the
|
||||||
|
GENERATOR's real prompt, so the two can never drift (the drill-down
|
||||||
|
E2E asserts on this exact template)."""
|
||||||
|
system, user = build_folder_summary_prompt(
|
||||||
|
"Homelab", "deployments/ansible",
|
||||||
|
[("Homelab", "deployments/ansible/lab-inventory.md", "Lab Inventory", None)],
|
||||||
|
)
|
||||||
|
assert compose_answer(_mock_body(system, user)) == (
|
||||||
|
"Fixture folder summary for Homelab/deployments/ansible."
|
||||||
|
)
|
||||||
|
# The source-root row names the source itself.
|
||||||
|
system, user = build_folder_summary_prompt("Homelab", "",
|
||||||
|
[("Homelab", "top.md", "Top", None)])
|
||||||
|
assert compose_answer(_mock_body(system, user)) == (
|
||||||
|
"Fixture folder summary for Homelab."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_mock_folder_marker_is_not_shadowed_by_the_summary_branch() -> None:
|
||||||
|
"""``FOLDER_SUMMARY_MODE`` contains ``SUMMARY_MODE`` as a substring —
|
||||||
|
the mock must check the folder branch FIRST, or every folder call
|
||||||
|
would land in the document-summary digest (regression pin)."""
|
||||||
|
system, user = build_folder_summary_prompt(
|
||||||
|
"S", "a", [("S", "a/x.md", "X", None)]
|
||||||
|
)
|
||||||
|
assert "SUMMARY_MODE" in system # the shadowing hazard is real
|
||||||
|
answer = compose_answer(_mock_body(system, user))
|
||||||
|
assert answer == "Fixture folder summary for S/a."
|
||||||
|
assert not answer.startswith("This document covers")
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_folder_happy_path_returns_trimmed_text() -> None:
|
||||||
|
docs = [("S", "a/x.md", "X", None)]
|
||||||
|
llm = _FakeLLM(reply=f" {REPLY} \n")
|
||||||
|
out = asyncio.run(summarize_folder("S", "a", docs, llm))
|
||||||
|
assert out == REPLY # the model's text, trimmed
|
||||||
|
assert llm.calls == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_folder_calls_the_configured_summary_model_with_marker() -> None:
|
||||||
|
docs = [("S", "a/x.md", "X", "X lead.")]
|
||||||
|
llm = _FakeLLM()
|
||||||
|
asyncio.run(summarize_folder("S", "a", docs, llm))
|
||||||
|
assert llm.model == llm.settings.llm_summary_model # the ``lite`` default
|
||||||
|
assert llm.model == "lite"
|
||||||
|
system, user = llm.requests[0]
|
||||||
|
assert FOLDER_SUMMARY_MODE in system
|
||||||
|
assert user.startswith(FOLDER_HEADER_PREFIX + "S/a")
|
||||||
|
assert "a/x.md — X — X lead." in user
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_folder_empty_reply_raises_llm_error() -> None:
|
||||||
|
docs = [("S", "a/x.md", "X", None)]
|
||||||
|
for reply in ("", " \n\t "):
|
||||||
|
llm = _FakeLLM(reply=reply)
|
||||||
|
with pytest.raises(LLMError, match="empty content for S/a"):
|
||||||
|
asyncio.run(summarize_folder("S", "a", docs, llm))
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_folder_error_propagates() -> None:
|
||||||
|
docs = [("S", "a/x.md", "X", None)]
|
||||||
|
llm = _FakeLLM(fail=LLMError("simulated transport failure"))
|
||||||
|
with pytest.raises(LLMError, match="simulated transport failure"):
|
||||||
|
asyncio.run(summarize_folder("S", "a", docs, llm))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- generate_folder_summaries (real Postgres) ----------
|
||||||
|
|
||||||
|
|
||||||
|
def _add_doc(
|
||||||
|
db: Session, source: str, path: str, title: str, summary: str | None = None
|
||||||
|
) -> Document:
|
||||||
|
doc = Document(
|
||||||
|
id=uuid.uuid4(),
|
||||||
|
source=source,
|
||||||
|
path=path,
|
||||||
|
full_path=f"/tmp/{path}",
|
||||||
|
title=title,
|
||||||
|
content="body",
|
||||||
|
content_hash="0" * 64,
|
||||||
|
summary=summary,
|
||||||
|
)
|
||||||
|
db.add(doc)
|
||||||
|
db.commit()
|
||||||
|
return doc
|
||||||
|
|
||||||
|
|
||||||
|
def _truncate(db: Session) -> None:
|
||||||
|
db.execute(text("TRUNCATE chunks, documents"))
|
||||||
|
db.execute(text("DELETE FROM folder_summaries"))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _rows(db: Session) -> dict[tuple[str, str], str]:
|
||||||
|
"""The stored folder summaries: ``{(source, folder_path): summary}``."""
|
||||||
|
result = db.execute(
|
||||||
|
text("SELECT source, folder_path, summary FROM folder_summaries")
|
||||||
|
).all()
|
||||||
|
return {(source, folder_path): summary for source, folder_path, summary in result}
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_catalogue(db: Session) -> None:
|
||||||
|
"""The shared catalogue: FSU has four docs in three candidate
|
||||||
|
folders (root 4, a 3, a/b 2 — all ≥ the minimum); FSU-solo has one
|
||||||
|
doc (its root folder is below the minimum — no row, no call)."""
|
||||||
|
_add_doc(db, "FSU", "a/b/one.md", "One", "One lead.\nSource: FSU/a/b/one.md")
|
||||||
|
_add_doc(db, "FSU", "a/b/two.md", "Two")
|
||||||
|
_add_doc(db, "FSU", "a/three.md", "Three")
|
||||||
|
_add_doc(db, "FSU", "root.md", "Root")
|
||||||
|
_add_doc(db, "FSU-solo", "solo.md", "Solo")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def clean_tables(db: Session):
|
||||||
|
_truncate(db)
|
||||||
|
yield
|
||||||
|
_truncate(db)
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_happy_path_upserts_every_candidate_folder(
|
||||||
|
db: Session, clean_tables, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""Every folder with ≥ 2 recursive docs gets a row (the source root
|
||||||
|
row included — ``folder_path = ''``); single-doc folders get none;
|
||||||
|
rows are stamped fresh; the stats dict and the log line are right;
|
||||||
|
folders are processed in deterministic (source, folder_path) order."""
|
||||||
|
_seed_catalogue(db)
|
||||||
|
llm = _FakeLLM()
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.rag.folder_summaries"):
|
||||||
|
stats = asyncio.run(generate_folder_summaries(db, llm))
|
||||||
|
assert stats == {"generated": 3, "failed": 0, "pruned": 0}
|
||||||
|
assert llm.calls == 3, "one lite call per candidate folder (the solo folder: none)"
|
||||||
|
|
||||||
|
stored = _rows(db)
|
||||||
|
assert set(stored) == {("FSU", ""), ("FSU", "a"), ("FSU", "a/b")}
|
||||||
|
assert all(summary == REPLY for summary in stored.values())
|
||||||
|
assert ("FSU-solo", "") not in stored, (
|
||||||
|
"a single-doc folder is fully described by its one file line — no row"
|
||||||
|
)
|
||||||
|
|
||||||
|
row = db.get(FolderSummary, ("FSU", "a/b"))
|
||||||
|
assert row is not None
|
||||||
|
assert row.summary == REPLY
|
||||||
|
assert row.updated_at is not None
|
||||||
|
age = datetime.now(UTC) - row.updated_at
|
||||||
|
assert age.total_seconds() < 300, "updated_at must be a fresh UTC timestamp"
|
||||||
|
|
||||||
|
# Deterministic (source, folder_path) order — root before the
|
||||||
|
# nested folders, one header per call.
|
||||||
|
assert [user.splitlines()[0] for _s, user in llm.requests] == [
|
||||||
|
"Folder: FSU",
|
||||||
|
"Folder: FSU/a",
|
||||||
|
"Folder: FSU/a/b",
|
||||||
|
]
|
||||||
|
# The recursive-subtree input: the a/ prompt carries a/b's docs too.
|
||||||
|
a_prompt = llm.requests[1][1]
|
||||||
|
assert "a/b/one.md — One — One lead." in a_prompt
|
||||||
|
assert "a/three.md — Three" in a_prompt
|
||||||
|
assert "root.md — Root" not in a_prompt
|
||||||
|
|
||||||
|
assert (
|
||||||
|
"folder_summaries: generated=3 failed=0 pruned=0" in caplog.text
|
||||||
|
), "the stats line must be greppable (PLAN §9 ample logging)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_per_folder_fail_soft_keeps_previous_and_lands_others(
|
||||||
|
db: Session, clean_tables, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""One folder's lite failure is logged and counted, its PREVIOUS
|
||||||
|
row is kept (an old summary is better than none), and the remaining
|
||||||
|
folders still land — a lite outage never fails the sync."""
|
||||||
|
_seed_catalogue(db)
|
||||||
|
db.add(FolderSummary(source="FSU", folder_path="a/b", summary="old summary"))
|
||||||
|
db.commit()
|
||||||
|
llm = _FakeLLM(fail_folders=("FSU/a/b",))
|
||||||
|
with caplog.at_level(logging.ERROR, logger="app.rag.folder_summaries"):
|
||||||
|
stats = asyncio.run(generate_folder_summaries(db, llm))
|
||||||
|
assert stats == {"generated": 2, "failed": 1, "pruned": 0}
|
||||||
|
assert llm.calls == 3 # the failing folder was attempted too
|
||||||
|
|
||||||
|
stored = _rows(db)
|
||||||
|
assert stored[("FSU", "a/b")] == "old summary", (
|
||||||
|
"the previous row survives the per-folder failure"
|
||||||
|
)
|
||||||
|
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a")] == REPLY, (
|
||||||
|
"the other folders still land"
|
||||||
|
)
|
||||||
|
assert "folder summary failed for FSU/a/b" in caplog.text
|
||||||
|
assert "simulated lite-model failure for FSU/a/b" in caplog.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_per_folder_fail_soft_without_previous_row_creates_nothing(
|
||||||
|
db: Session, clean_tables
|
||||||
|
) -> None:
|
||||||
|
_seed_catalogue(db)
|
||||||
|
llm = _FakeLLM(fail_folders=("FSU/a/b",))
|
||||||
|
stats = asyncio.run(generate_folder_summaries(db, llm))
|
||||||
|
assert stats["failed"] == 1
|
||||||
|
stored = _rows(db)
|
||||||
|
assert ("FSU", "a/b") not in stored, "no row must be invented for a failed folder"
|
||||||
|
assert ("FSU", "") in stored and ("FSU", "a") in stored
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_prunes_stale_rows_and_keeps_live_ones(db: Session, clean_tables) -> None:
|
||||||
|
"""Rows for folders that dropped below 2 recursive docs are deleted
|
||||||
|
(pruned/renamed — the summary would go stale); rows for folders
|
||||||
|
that still qualify persist (an unchanged folder's summary is still
|
||||||
|
true — regenerated in place)."""
|
||||||
|
_seed_catalogue(db)
|
||||||
|
# A stale row for a folder no longer in the catalogue (3→1 docs /
|
||||||
|
# renamed away) + a live row with old content.
|
||||||
|
db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale"))
|
||||||
|
db.add(FolderSummary(source="FSU", folder_path="a", summary="old a summary"))
|
||||||
|
db.add(FolderSummary(source="FSU-solo", folder_path="", summary="solo stale"))
|
||||||
|
db.commit()
|
||||||
|
stats = asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||||
|
assert stats["pruned"] == 2 # gone/old + the FSU-solo root (1 doc)
|
||||||
|
|
||||||
|
stored = _rows(db)
|
||||||
|
assert ("FSU", "gone/old") not in stored, "the stale folder row must be pruned"
|
||||||
|
assert ("FSU-solo", "") not in stored, (
|
||||||
|
"a folder that dropped below 2 docs loses its row"
|
||||||
|
)
|
||||||
|
assert ("FSU", "a") in stored, "the still-qualifying folder keeps its row"
|
||||||
|
assert stored[("FSU", "a")] == REPLY # regenerated, not stale
|
||||||
|
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a/b")] == REPLY
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_skip_is_a_full_noop(db: Session, clean_tables) -> None:
|
||||||
|
"""``skip=True`` (the ``--limit`` debug run): the LLM is never
|
||||||
|
called, no rows are touched, zero stats."""
|
||||||
|
_seed_catalogue(db)
|
||||||
|
db.add(FolderSummary(source="FSU", folder_path="", summary="existing"))
|
||||||
|
db.commit()
|
||||||
|
llm = _FakeLLM()
|
||||||
|
stats = asyncio.run(generate_folder_summaries(db, llm, skip=True))
|
||||||
|
assert stats == {"generated": 0, "failed": 0, "pruned": 0}
|
||||||
|
assert llm.calls == 0
|
||||||
|
assert _rows(db) == {("FSU", ""): "existing"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_empty_kb_prunes_every_row(db: Session, clean_tables) -> None:
|
||||||
|
"""No documents → no candidate folders → every stored row is
|
||||||
|
pruned, with zero wasted lite calls."""
|
||||||
|
db.add(FolderSummary(source="FSU", folder_path="", summary="old"))
|
||||||
|
db.add(FolderSummary(source="FSU", folder_path="a/b", summary="old"))
|
||||||
|
db.commit()
|
||||||
|
llm = _FakeLLM()
|
||||||
|
stats = asyncio.run(generate_folder_summaries(db, llm))
|
||||||
|
assert stats == {"generated": 0, "failed": 0, "pruned": 2}
|
||||||
|
assert llm.calls == 0
|
||||||
|
assert _rows(db) == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_summarizes_folder_whose_prefix_is_also_a_doc_path(
|
||||||
|
db: Session, clean_tables
|
||||||
|
) -> None:
|
||||||
|
"""The ``path == folder`` arm end to end: a file sharing its name
|
||||||
|
with a directory counts toward the folder's recursive count (2
|
||||||
|
docs → the folder is summarized, and BOTH docs are in its prompt).
|
||||||
|
"""
|
||||||
|
_add_doc(db, "FSU", "a/b", "B") # a file named "b" (its path is a prefix)
|
||||||
|
_add_doc(db, "FSU", "a/b/c.md", "C") # and a real folder "a/b/"
|
||||||
|
llm = _FakeLLM()
|
||||||
|
stats = asyncio.run(generate_folder_summaries(db, llm))
|
||||||
|
assert stats["generated"] == 3 # root (2), a (2), a/b (2) — all ≥ the minimum
|
||||||
|
stored = _rows(db)
|
||||||
|
assert set(stored) == {("FSU", ""), ("FSU", "a"), ("FSU", "a/b")}
|
||||||
|
a_b_prompt = [
|
||||||
|
user for _system, user in llm.requests if user.startswith("Folder: FSU/a/b\n")
|
||||||
|
][0]
|
||||||
|
assert "a/b — B" in a_b_prompt
|
||||||
|
assert "a/b/c.md — C" in a_b_prompt
|
||||||
|
|
||||||
|
|
||||||
|
def test_generate_only_flushes_caller_commits(db: Session, clean_tables) -> None:
|
||||||
|
"""The generator only flushes — the sync path owns the transaction
|
||||||
|
(the phase-53 ``bump_sources_version`` convention): the catalogue
|
||||||
|
is committed (the real sync path commits the import before the
|
||||||
|
summary hooks run), but a second session sees the generator's rows
|
||||||
|
as NOTHING until the CALLER commits — and sees them after."""
|
||||||
|
_add_doc(db, "FSU", "x/y/one.md", "One")
|
||||||
|
_add_doc(db, "FSU", "x/y/two.md", "Two")
|
||||||
|
stats = asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||||
|
assert stats["generated"] == 3 # root + x + x/y — all 2 recursive docs
|
||||||
|
|
||||||
|
with SessionLocal() as other:
|
||||||
|
n = other.scalar(
|
||||||
|
text("SELECT count(*) FROM folder_summaries WHERE source = 'FSU'")
|
||||||
|
)
|
||||||
|
assert n == 0, "unflushed-by-caller rows must not be visible yet"
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
with SessionLocal() as other:
|
||||||
|
n = other.scalar(
|
||||||
|
text("SELECT count(*) FROM folder_summaries WHERE source = 'FSU'")
|
||||||
|
)
|
||||||
|
assert n == 3, "the caller's commit makes the flushed rows durable"
|
||||||
|
|
||||||
|
assert MIN_DOCS_PER_FOLDER == 2 # the ≥ 2 scope rule, pinned by name
|
||||||
|
|
||||||
|
|
||||||
|
def test_folder_summary_table_empty_gate(db: Session, clean_tables) -> None:
|
||||||
|
"""The sync-path gate probe (phase 94, task 02): empty → True
|
||||||
|
(the first full sync after migration 0017 must still generate),
|
||||||
|
one row → False (a populated table waits for a KB change)."""
|
||||||
|
assert folder_summary_table_empty(db) is True # the truncated table
|
||||||
|
_add_doc(db, "FSU", "a/one.md", "One")
|
||||||
|
_add_doc(db, "FSU", "a/two.md", "Two")
|
||||||
|
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||||
|
db.commit()
|
||||||
|
assert folder_summary_table_empty(db) is False # rows landed
|
||||||
|
db.execute(text("DELETE FROM folder_summaries"))
|
||||||
|
db.commit()
|
||||||
|
assert folder_summary_table_empty(db) is True # emptied again
|
||||||
@@ -4,12 +4,18 @@ The mock (``tests/e2e/mock_llm.py``) classifies marker requests
|
|||||||
statelessly into one step of the agent tool flow. This file pins the
|
statelessly into one step of the agent tool flow. This file pins the
|
||||||
classification at unit speed — no Playwright, no LLM process:
|
classification at unit speed — no Playwright, no LLM process:
|
||||||
|
|
||||||
* the phase-37 SINGLE-READ flow (``TOOLS_TRIGGER`` only) stays
|
* the phase-37/94 SINGLE-READ flow (``TOOLS_TRIGGER`` only): list (the
|
||||||
byte-identical: list → read (first catalog line, ``call_1``) → answer;
|
top-level source listing) → drill (``ls`` scoped to the first source
|
||||||
* the phase-45 MULTI-READ flow (``TOOLS_TRIGGER`` + ``MULTI_READ_TRIGGER``)
|
— phase 94: the top level carries sources only, so the flow drills
|
||||||
classifies by the count of ``tool``-role read results: list → read #1
|
one level for the file lines) → read (first file line, the combined
|
||||||
(``call_1``) → read #2 (second catalog line, ``call_2``) → the
|
``source/path``) → answer;
|
||||||
byte-stable ``multi_answer`` naming both read paths.
|
* the phase-45 MULTI-READ flow (``TOOLS_TRIGGER`` +
|
||||||
|
``MULTI_READ_TRIGGER``) classifies by the count of ``tool``-role read
|
||||||
|
results: list → drill → read #1 (first file line) → read #2 (second
|
||||||
|
file line) → the byte-stable ``multi_answer`` naming both read paths;
|
||||||
|
* the degenerate empty-KB case: every listed source already drilled
|
||||||
|
with no file lines → the flow falls back to the re-list loop (the
|
||||||
|
round cap settles it — the phase-70 empty-catalog behavior).
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -36,28 +42,50 @@ SYSTEM_LOW = "<relevance>LOW</relevance>\n"
|
|||||||
#: the phase-70 harness-aligned names).
|
#: the phase-70 harness-aligned names).
|
||||||
TOOLS = [{"type": "function", "function": {"name": "ls"}}]
|
TOOLS = [{"type": "function", "function": {"name": "ls"}}]
|
||||||
|
|
||||||
#: The agent's ``ls`` output for a two-document KB
|
#: The agent's drill-down ``ls`` output for a two-source KB
|
||||||
#: (``app/rag/agent.py`` ``_execute_tool``): one
|
#: (``app/rag/agent.py`` ``render_ls_top`` / ``render_folder_listing``,
|
||||||
#: ``source: X | path: Y | title: Z`` line per document (phase 63: labeled,
|
#: phase 94): the top level lists the registered sources (registry
|
||||||
#: unambiguous fields), ``(source, path)`` order.
|
#: order, recursive counts — no file lines); the folder level carries
|
||||||
CATALOG_2 = (
|
#: the file lines (``source: X | path: Y | title: Z`` — the phase-63
|
||||||
"2 documents:\n"
|
#: labeled fields, unchanged), ``path`` order.
|
||||||
"source: Deployments | path: example-record-file.json | title: Example Record File\n"
|
TOP_LEVEL_2 = (
|
||||||
"source: Homelab | path: aws-route53.md | title: AWS Route 53 Notes"
|
"2 sources:\n"
|
||||||
|
"\n"
|
||||||
|
"Deployments — 1 documents\n"
|
||||||
|
"Homelab — 1 documents"
|
||||||
)
|
)
|
||||||
|
|
||||||
CATALOG_1 = (
|
#: The first source's root folder: one file line (the single-read
|
||||||
"1 documents:\n"
|
#: flow's read target).
|
||||||
|
FOLDER_DEPLOYMENTS = (
|
||||||
|
"Deployments — 1 documents, 0 folders:\n"
|
||||||
|
"\n"
|
||||||
"source: Deployments | path: example-record-file.json | title: Example Record File"
|
"source: Deployments | path: example-record-file.json | title: Example Record File"
|
||||||
)
|
)
|
||||||
|
|
||||||
CATALOG_3 = (
|
#: Two file lines in the first source (the multi-read flow's reads).
|
||||||
"3 documents:\n"
|
FOLDER_DEPLOYMENTS_2 = (
|
||||||
|
"Deployments — 2 documents, 0 folders:\n"
|
||||||
|
"\n"
|
||||||
|
"source: Deployments | path: aaa.md | title: AAA\n"
|
||||||
|
"source: Deployments | path: bbb.md | title: BBB"
|
||||||
|
)
|
||||||
|
|
||||||
|
#: Three file lines in the first source (the listing-order pin: read
|
||||||
|
#: #2 is the SECOND line, not the last).
|
||||||
|
FOLDER_DEPLOYMENTS_3 = (
|
||||||
|
"Deployments — 3 documents, 0 folders:\n"
|
||||||
|
"\n"
|
||||||
"source: Deployments | path: aaa.md | title: AAA\n"
|
"source: Deployments | path: aaa.md | title: AAA\n"
|
||||||
"source: Deployments | path: bbb.md | title: BBB\n"
|
"source: Deployments | path: bbb.md | title: BBB\n"
|
||||||
"source: Homelab | path: ccc.md | title: CCC"
|
"source: Deployments | path: ccc.md | title: CCC"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
#: Empty folder levels (a registered source with no documents — the
|
||||||
|
#: header line alone; the drill's degenerate arm).
|
||||||
|
FOLDER_DEPLOYMENTS_EMPTY = "Deployments — 0 documents, 0 folders:"
|
||||||
|
FOLDER_HOMELAB_EMPTY = "Homelab — 0 documents, 0 folders:"
|
||||||
|
|
||||||
DOC1_SP = "Deployments/example-record-file.json"
|
DOC1_SP = "Deployments/example-record-file.json"
|
||||||
DOC1_CONTENT = (
|
DOC1_CONTENT = (
|
||||||
"The record file keeps every hosted zone record — first line is longer "
|
"The record file keeps every hosted zone record — first line is longer "
|
||||||
@@ -122,10 +150,39 @@ def test_single_flow_list_step() -> None:
|
|||||||
assert _tool_flow(_body(SINGLE_USER)) == ("list", "", "")
|
assert _tool_flow(_body(SINGLE_USER)) == ("list", "", "")
|
||||||
|
|
||||||
|
|
||||||
def test_single_flow_read_step_first_catalog_line() -> None:
|
def test_single_flow_drill_step_after_top_level() -> None:
|
||||||
flow = _tool_flow(_body(SINGLE_USER, (CATALOG_3,)))
|
# Phase 94: the top level lists SOURCES only — the flow drills one
|
||||||
# The FIRST listing line (Deployments/aaa.md), labeled fields.
|
# level into the FIRST source (listing order = registry order).
|
||||||
assert flow == ("read", "Deployments", "aaa.md", "call_1")
|
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2,)))
|
||||||
|
assert flow == ("drill", "Deployments", "call_1")
|
||||||
|
|
||||||
|
|
||||||
|
def test_single_flow_drill_skips_already_drilled_source() -> None:
|
||||||
|
# The first source's folder level is already in the messages (an
|
||||||
|
# empty listing — header only, no file lines): the drill proceeds
|
||||||
|
# to the NEXT un-drilled source.
|
||||||
|
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS_EMPTY)))
|
||||||
|
assert flow == ("drill", "Homelab", "call_1")
|
||||||
|
|
||||||
|
|
||||||
|
def test_single_flow_all_sources_drilled_empty_falls_back_to_list() -> None:
|
||||||
|
# Degenerate: every listed source already drilled, no file lines
|
||||||
|
# anywhere — the flow falls back to the re-list loop (settled at the
|
||||||
|
# round cap, the phase-70 empty-catalog behavior).
|
||||||
|
flow = _tool_flow(
|
||||||
|
_body(
|
||||||
|
SINGLE_USER,
|
||||||
|
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_EMPTY, FOLDER_HOMELAB_EMPTY),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert flow == ("list", "", "")
|
||||||
|
|
||||||
|
|
||||||
|
def test_single_flow_read_step_first_file_line() -> None:
|
||||||
|
# The folder level reached: the FIRST file line (Deployments/aaa.md),
|
||||||
|
# labeled fields, the combined ``source/path`` join.
|
||||||
|
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS_2)))
|
||||||
|
assert flow == ("read", "Deployments", "aaa.md", "call_2")
|
||||||
|
|
||||||
|
|
||||||
def test_read_step_nested_path_stays_intact() -> None:
|
def test_read_step_nested_path_stays_intact() -> None:
|
||||||
@@ -133,19 +190,20 @@ def test_read_step_nested_path_stays_intact() -> None:
|
|||||||
# ``source/path — title`` + ``rpartition("/")`` parse misread the
|
# ``source/path — title`` + ``rpartition("/")`` parse misread the
|
||||||
# split (``source=brain-of-reese-main/homelab``). The labeled fields
|
# split (``source=brain-of-reese-main/homelab``). The labeled fields
|
||||||
# recover the nested path intact, however deep.
|
# recover the nested path intact, however deep.
|
||||||
catalog = (
|
listing = (
|
||||||
"1 documents:\n"
|
"brain-of-reese-main — 1 documents, 0 folders:\n"
|
||||||
|
"\n"
|
||||||
"source: brain-of-reese-main | path: homelab/aws-route53.md | title: aws-route53"
|
"source: brain-of-reese-main | path: homelab/aws-route53.md | title: aws-route53"
|
||||||
)
|
)
|
||||||
flow = _tool_flow(_body(SINGLE_USER, (catalog,)))
|
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, listing)))
|
||||||
assert flow == ("read", "brain-of-reese-main", "homelab/aws-route53.md", "call_1")
|
assert flow == ("read", "brain-of-reese-main", "homelab/aws-route53.md", "call_2")
|
||||||
|
|
||||||
|
|
||||||
def test_single_flow_answer_step_with_tools_offered() -> None:
|
def test_single_flow_answer_step_with_tools_offered() -> None:
|
||||||
# Phase 45: the round cap keeps the tools offered until it is hit —
|
# Phase 45: the round cap keeps the tools offered until it is hit —
|
||||||
# the answer step fires regardless of the ``tools`` parameter.
|
# the answer step fires regardless of the ``tools`` parameter.
|
||||||
flow = _tool_flow(
|
flow = _tool_flow(
|
||||||
_body(SINGLE_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT)))
|
_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
|
||||||
)
|
)
|
||||||
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
|
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
|
||||||
|
|
||||||
@@ -154,7 +212,7 @@ def test_single_flow_answer_step_without_tools() -> None:
|
|||||||
flow = _tool_flow(
|
flow = _tool_flow(
|
||||||
_body(
|
_body(
|
||||||
SINGLE_USER,
|
SINGLE_USER,
|
||||||
(CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT)),
|
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)),
|
||||||
tools=None,
|
tools=None,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -184,22 +242,39 @@ def test_multi_flow_list_step() -> None:
|
|||||||
assert _tool_flow(_body(MULTI_USER)) == ("list", "", "")
|
assert _tool_flow(_body(MULTI_USER)) == ("list", "", "")
|
||||||
|
|
||||||
|
|
||||||
|
def test_multi_flow_drill_step_after_top_level() -> None:
|
||||||
|
# Phase 94: the top level lists SOURCES only — the multi flow drills
|
||||||
|
# too, before its first read.
|
||||||
|
flow = _tool_flow(_body(MULTI_USER, (TOP_LEVEL_2,)))
|
||||||
|
assert flow == ("drill", "Deployments", "call_1")
|
||||||
|
|
||||||
|
|
||||||
def test_multi_flow_read_first_step() -> None:
|
def test_multi_flow_read_first_step() -> None:
|
||||||
flow = _tool_flow(_body(MULTI_USER, (CATALOG_2,)))
|
flow = _tool_flow(_body(MULTI_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS)))
|
||||||
assert flow == ("read", DOC1_SP.split("/", 1)[0], DOC1_SP.rsplit("/", 1)[1], "call_1")
|
assert flow == ("read", DOC1_SP.split("/", 1)[0], DOC1_SP.rsplit("/", 1)[1], "call_2")
|
||||||
|
|
||||||
|
|
||||||
def test_multi_flow_read_second_step_skips_already_read() -> None:
|
def test_multi_flow_read_second_step_skips_already_read() -> None:
|
||||||
flow = _tool_flow(_body(MULTI_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT))))
|
flow = _tool_flow(
|
||||||
# The second catalog line — the first line differing from DOC1.
|
_body(
|
||||||
assert flow == ("read", "Homelab", "aws-route53.md", "call_2")
|
MULTI_USER,
|
||||||
|
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_2, _read_result("Deployments/aaa.md", DOC1_CONTENT)),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
# The second file line — the first line differing from the read doc.
|
||||||
|
assert flow == ("read", "Deployments", "bbb.md", "call_3")
|
||||||
|
|
||||||
|
|
||||||
def test_multi_flow_read_second_is_listing_order_not_last() -> None:
|
def test_multi_flow_read_second_is_listing_order_not_last() -> None:
|
||||||
# Three-doc catalog, first doc read: read #2 is the SECOND line
|
# Three-file listing, first file read: read #2 is the SECOND line
|
||||||
# (Deployments/bbb.md), not the last one.
|
# (Deployments/bbb.md), not the last one.
|
||||||
flow = _tool_flow(_body(MULTI_USER, (CATALOG_3, _read_result("Deployments/aaa.md", "x"))))
|
flow = _tool_flow(
|
||||||
assert flow == ("read", "Deployments", "bbb.md", "call_2")
|
_body(
|
||||||
|
MULTI_USER,
|
||||||
|
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_3, _read_result("Deployments/aaa.md", "x")),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert flow == ("read", "Deployments", "bbb.md", "call_3")
|
||||||
|
|
||||||
|
|
||||||
def test_multi_flow_answer_step_names_both_paths() -> None:
|
def test_multi_flow_answer_step_names_both_paths() -> None:
|
||||||
@@ -207,7 +282,8 @@ def test_multi_flow_answer_step_names_both_paths() -> None:
|
|||||||
_body(
|
_body(
|
||||||
MULTI_USER,
|
MULTI_USER,
|
||||||
(
|
(
|
||||||
CATALOG_2,
|
TOP_LEVEL_2,
|
||||||
|
FOLDER_DEPLOYMENTS_2,
|
||||||
_read_result(DOC1_SP, DOC1_CONTENT),
|
_read_result(DOC1_SP, DOC1_CONTENT),
|
||||||
_read_result(DOC2_SP, DOC2_CONTENT),
|
_read_result(DOC2_SP, DOC2_CONTENT),
|
||||||
),
|
),
|
||||||
@@ -227,7 +303,8 @@ def test_multi_flow_answer_step_without_tools_offered() -> None:
|
|||||||
_body(
|
_body(
|
||||||
MULTI_USER,
|
MULTI_USER,
|
||||||
(
|
(
|
||||||
CATALOG_2,
|
TOP_LEVEL_2,
|
||||||
|
FOLDER_DEPLOYMENTS_2,
|
||||||
_read_result(DOC1_SP, DOC1_CONTENT),
|
_read_result(DOC1_SP, DOC1_CONTENT),
|
||||||
_read_result(DOC2_SP, DOC2_CONTENT),
|
_read_result(DOC2_SP, DOC2_CONTENT),
|
||||||
),
|
),
|
||||||
@@ -238,11 +315,11 @@ def test_multi_flow_answer_step_without_tools_offered() -> None:
|
|||||||
assert flow[0] == "multi_answer"
|
assert flow[0] == "multi_answer"
|
||||||
|
|
||||||
|
|
||||||
def test_multi_flow_one_document_catalog_degenerates_to_single_answer() -> None:
|
def test_multi_flow_one_file_listing_degenerates_to_single_answer() -> None:
|
||||||
# Nothing second to read — the single-read answer shape, quoting the
|
# Nothing second to read — the single-read answer shape, quoting the
|
||||||
# only read result.
|
# only read result.
|
||||||
flow = _tool_flow(
|
flow = _tool_flow(
|
||||||
_body(MULTI_USER, (CATALOG_1, _read_result(DOC1_SP, DOC1_CONTENT)))
|
_body(MULTI_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
|
||||||
)
|
)
|
||||||
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
|
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
|
||||||
|
|
||||||
@@ -320,9 +397,12 @@ def test_search_flow_found_step_without_tools_offered() -> None:
|
|||||||
|
|
||||||
|
|
||||||
def test_search_flow_ignores_catalog_and_read_results() -> None:
|
def test_search_flow_ignores_catalog_and_read_results() -> None:
|
||||||
# A catalog (labeled lines) and a read result ("Document …" prefix)
|
# Listings (top-level + folder level, labeled file lines) and a read
|
||||||
# are NOT search results — the flow stays at the search step.
|
# result ("Document …" prefix) are NOT search results — the flow
|
||||||
flow = _search_flow(_body(SEARCH_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT))))
|
# stays at the search step.
|
||||||
|
flow = _search_flow(
|
||||||
|
_body(SEARCH_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
|
||||||
|
)
|
||||||
assert flow == ("search",)
|
assert flow == ("search",)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+30
-22
@@ -11,11 +11,15 @@ And the phase-71 deflection plain-text line (owner-permitted
|
|||||||
line; the ``DEFLECT_MODE`` marker-keying contract is unchanged and
|
line; the ``DEFLECT_MODE`` marker-keying contract is unchanged and
|
||||||
the line never leaks into the HIGH prompt.
|
the line never leaks into the HIGH prompt.
|
||||||
|
|
||||||
And the phase-72 ``<tools>`` copy: the document-identity contract is
|
And the ``<tools>`` copy (phase 72: the document-identity contract
|
||||||
stated up front (the ``ls`` source-name scope, the combined
|
stated up front — the combined ``source/path`` identity for
|
||||||
``source/path`` identity for ``read``/``grep``) — the same contract
|
``read``/``grep``; phase 94, task 03: the ``ls`` clause rewritten to
|
||||||
the teaching refusals in :mod:`app.rag.agent` re-state; the
|
the drill-down tree contract — one level per call, sources at the
|
||||||
``<tools>`` marker keying (HIGH only) is unchanged.
|
top, folders + files below, ``grep`` as the without-listing locator —
|
||||||
|
while the ``read``/``grep`` clauses and the discipline rules are
|
||||||
|
byte-identical): the teaching refusals in :mod:`app.rag.agent`
|
||||||
|
re-state the same contract; the ``<tools>`` marker keying (HIGH
|
||||||
|
only) is unchanged.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -194,20 +198,22 @@ pinned byte-for-byte in
|
|||||||
|
|
||||||
|
|
||||||
def test_tools_section_phase72_contract_clauses() -> None:
|
def test_tools_section_phase72_contract_clauses() -> None:
|
||||||
"""Phase 72: the two contract clauses the teaching refusals
|
"""Phase 72 + phase 94: the contract clauses the teaching refusals
|
||||||
re-state after the fact, pinned byte-for-byte in the constant —
|
re-state after the fact, pinned byte-for-byte in the constant —
|
||||||
the ``ls`` source-name clause (its optional ``path`` is a source
|
the ``ls`` clause (phase 94: the drill-down tree contract — one
|
||||||
name, not a directory or file path; omit it to list every
|
level per call, sources at the top, folders + files below, never
|
||||||
document) and the ``read``/``grep`` combined-identity clause
|
the whole KB in one call, ``grep`` as the without-listing locator)
|
||||||
(the combined ``source/path`` string exactly as shown in the
|
and the ``read``/``grep`` combined-identity clause (the combined
|
||||||
``ls`` output, *including the source name*; a bare document path
|
``source/path`` string exactly as shown in the ``ls`` output,
|
||||||
will not resolve)."""
|
*including the source name*; a bare document path will not
|
||||||
# The ls source-name clause.
|
resolve)."""
|
||||||
assert (
|
# The ls drill-down clauses (phase 94, task 03).
|
||||||
"a source name (e.g. 'homelab'), not a directory or file "
|
assert "one level at a time" in TOOLS_SECTION
|
||||||
"path — omit it to list every document"
|
assert "lists every synced source with its document count" in TOOLS_SECTION
|
||||||
) in TOOLS_SECTION
|
assert "that source's top-level folders and files" in TOOLS_SECTION
|
||||||
# The read combined-identity clause.
|
assert "never the whole knowledge base in one call" in TOOLS_SECTION
|
||||||
|
assert "to find one specific document without listing, use `grep`" in TOOLS_SECTION
|
||||||
|
# The read combined-identity clause (byte-identical across phases).
|
||||||
assert (
|
assert (
|
||||||
"combined `source/path` string, exactly as shown in the `ls` "
|
"combined `source/path` string, exactly as shown in the `ls` "
|
||||||
"output — including the source name"
|
"output — including the source name"
|
||||||
@@ -217,23 +223,25 @@ def test_tools_section_phase72_contract_clauses() -> None:
|
|||||||
"a bare document path (without the source name) will not resolve"
|
"a bare document path (without the source name) will not resolve"
|
||||||
) == 2
|
) == 2
|
||||||
# The pre-phase-70 scope wording is gone — replaced by the
|
# The pre-phase-70 scope wording is gone — replaced by the
|
||||||
# explicit source-name contract.
|
# explicit source-name contract (and the phase-72 source-name-only
|
||||||
|
# clause by the phase-94 drill-down contract).
|
||||||
assert "pass a source name as `path`" not in TOOLS_SECTION
|
assert "pass a source name as `path`" not in TOOLS_SECTION
|
||||||
|
assert "not a directory or file path" not in TOOLS_SECTION
|
||||||
|
|
||||||
|
|
||||||
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
|
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
|
||||||
"""Phase 72: the contract clauses ride the HIGH prompt with the
|
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
|
||||||
rest of the section and never leak into the LOW/deflection prompt
|
rest of the section and never leak into the LOW/deflection prompt
|
||||||
(whose byte-identity is pinned in
|
(whose byte-identity is pinned in
|
||||||
:func:`test_zero_note_prompt_is_byte_identical_to_pre_steering`)."""
|
:func:`test_zero_note_prompt_is_byte_identical_to_pre_steering`)."""
|
||||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||||
high = build_high_prompt([doc])
|
high = build_high_prompt([doc])
|
||||||
assert "<tools>" in high
|
assert "<tools>" in high
|
||||||
assert "not a directory or file path" in high
|
assert "one level at a time" in high # the phase-94 ls clause
|
||||||
assert "including the source name" in high
|
assert "including the source name" in high
|
||||||
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
|
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
|
||||||
assert "<tools>" not in low
|
assert "<tools>" not in low
|
||||||
assert "not a directory or file path" not in low
|
assert "one level at a time" not in low
|
||||||
assert "including the source name" not in low
|
assert "including the source name" not in low
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -752,8 +752,9 @@ def _patch_sync_seams(
|
|||||||
"""The runner's seams, monkeypatched on ``app.api.sync`` (the house
|
"""The runner's seams, monkeypatched on ``app.api.sync`` (the house
|
||||||
mock-import pattern): fresh settings (no ``.env`` leak), a no-op
|
mock-import pattern): fresh settings (no ``.env`` leak), a no-op
|
||||||
model probe, a sentinel LLM client, one git row, the gated clone +
|
model probe, a sentinel LLM client, one git row, the gated clone +
|
||||||
import, a no-op overview, and the DB-free sources-version step
|
import, a no-op overview, a no-op folder-summary step (phase 94 —
|
||||||
(dummy session + pinned counters)."""
|
the sentinel LLM client has no ``chat``), and the DB-free
|
||||||
|
sources-version step (dummy session + pinned counters)."""
|
||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
sync_api,
|
sync_api,
|
||||||
"get_settings",
|
"get_settings",
|
||||||
@@ -783,6 +784,13 @@ def _patch_sync_seams(
|
|||||||
|
|
||||||
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
||||||
|
|
||||||
|
async def fake_folder_summaries(
|
||||||
|
db: object, llm: object, *, skip: bool = False
|
||||||
|
) -> dict[str, int]:
|
||||||
|
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||||
|
|
||||||
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)
|
||||||
|
|
||||||
class _DummySession:
|
class _DummySession:
|
||||||
def close(self) -> None:
|
def close(self) -> None:
|
||||||
pass
|
pass
|
||||||
|
|||||||
Reference in New Issue
Block a user