phase: 94_ls_tree_drilldown
All green. Verification complete. **Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)** - Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal - Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths - Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met - `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %) - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched) - Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed - Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol) **Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
# Phase 94 — `ls` becomes a drill-down tree with sync-time folder summaries
|
||||
|
||||
**Source:** `TODO.md` L4 — "Brain of reese can end up indexing thousands of files. When the llm calls `list` it can end up filling the context entirely with just the document lists. I think list should work like a filesystem tree where it shows only the current "folder" (or at the top level, the list of synced projects) and then allow the LLM to drill down into the tree structure as it needs. We should add a level of "smart" to this by summarizing what each folder contains (at sync time) so when the LLM calls list it gets a summary of what's in each directory. Run tests against this new architecture and ensure accuracy and performance aren't too badly impacted."
|
||||
**Story:** n/a (owner TODO item — agent tooling on `37_agent_document_tools` / `45_agent_unlimited_tools` / `68_search_tool` / `70_harness_aligned_tools`).
|
||||
**Context:** Today `ls` (the harness-aligned surface, phase 70 — `app/rag/agent.py` `AGENT_TOOLS`) lists **every** indexed document as `source: X | path: Y | title: Z` lines (or one source's documents when `path` is a source name) — with a 1,340-document KB that is the entire context in one call. `read`/`grep` take the combined `source/path` identity; `grep` is the whole-KB locator. The one-shot `lite` machinery exists: document summaries (`app/rag/summarizer.py`, `SUMMARY_MODE`) and the KB overview (`app/rag/overview.py`, `KB_OVERVIEW_MODE`) — both change-gated at import, best-effort/fail-soft, with a marker the deterministic E2E mock (`tests/e2e/mock_llm.py`) keys on. Sync paths: `scripts/import_docs.py` (`_run()`, the overview hook) and `POST /api/sync` (`app/api/sync.py`, step 5). The controlled accuracy/performance methodology is `TOOL_CALLING_TESTING.md` + `uv run python -m scripts.agent_realmodel_check --restore --mode fixture` (the 8-document, 2-source, **folder-structured** fixture KB in `tests/fixtures/test_kb.dump.sql`: `deployments/{ansible,ci,quadlet}` + `homelab/{backups,containers,networking}`).
|
||||
|
||||
## Objective
|
||||
Rebuild `ls` as a filesystem-style tree the LLM drills through: top level lists the synced projects (sources) with a per-source summary; `ls` of a source lists its folders (each with a sync-time summary) + top-level files; `ls` of a folder lists its subfolders + files. Folder summaries are generated at sync time by the `lite` model (change-gated, fail-soft). The controlled tool-calling battery then proves accuracy and performance are not badly impacted.
|
||||
|
||||
## Dependencies
|
||||
- `93_theme_semantic_completion` (todo) — queue order only; no code dependency (different subsystems).
|
||||
|
||||
## Owner-permitted decision recorded here (PLAN.md is being redone by the owner)
|
||||
- **Tool-surface revision (owner permission 2026-09-10, TODO.md L4):** the `ls` RESULT format and `path` semantics change (tree drill-down); the `ls` NAME and the `read`/`grep` contract (combined `source/path`) are untouched. This extends the phase-70 harness-aligned surface per the owner's explicit request; the prompt teaching (`TOOLS_SECTION`) is rewritten to match in the same change.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
### Tree shape (task 03)
|
||||
Document paths are already tree-like (`deployments/ansible/lab-inventory.md`); a **folder** is a path prefix. Three `ls` levels, one argument:
|
||||
|
||||
| Call | Result |
|
||||
|---|---|
|
||||
| `ls()` (no path) | every registered source (registry order, `list_source_names`): `name — N documents` + an indented summary line when one is stored (folders with no docs still list as `0 documents`) |
|
||||
| `ls(path=<source>)` | the source's **root folder**: each subfolder ` <folder>/ — N documents[: summary]` (count = documents with `path == folder` or `path startswith folder + "/"`), then the root's own file lines `source: X | path: Y | title: Z` |
|
||||
| `ls(path=<source>/<folder>[/…])` | that folder's subfolders (same shape, paths relative to the source) + the folder's own file lines |
|
||||
|
||||
- **File-line cap:** a folder's own files list at most **50** lines (path order — catalog order), then one deterministic note: `…and {N−50} more documents in this folder — use grep (pattern) to find a specific one.` (folds the old whole-KB flood into one bounded level; `grep` stays the locator — no new tool).
|
||||
- **Summaries:** shown for the SOURCE (top level) and for each SUBFOLDER, when stored; absent → the count line only (no placeholder).
|
||||
- **Refusals (adapted from the phase-70/72 teaching lines, same style — one line, argument echoed, counts in nothing, consumes a round):** unknown first segment → the existing no-source refusal; a `source/…` argument whose folder matches no indexed prefix → a NOT-A-FOLDER line teaching the drill-down (echo the argument, list the parent folder's subfolders so the model can self-correct in the next round). `ls` of a registered source with no documents keeps the `0 documents:` behavior.
|
||||
- **`holder.tool_calls`** increments on every successful listing (top/root/folder); refusals count in nothing (unchanged house rule).
|
||||
|
||||
### Folder summaries (tasks 01, 02)
|
||||
- **Storage:** new table `folder_summaries` — PK `(source, folder_path)` (column types mirror `Document.source` / `Document.path`), `summary` `Text`, `updated_at` `timestamptz`. `folder_path = ""` is the SOURCE ROOT (the top-level source summary); a folder is any other path prefix.
|
||||
- **Generation** (`app/rag/folder_summaries.py`, mirroring `app/rag/overview.py`): marker `FOLDER_SUMMARY_MODE` (the E2E mock keys on it — `tests/e2e/mock_llm.py` gains the canned branch); input per folder = its **recursive subtree** — every document whose path equals the folder or starts with `folder + "/"` (the same set the `ls` count shows; `folder_path ""` = the whole source), each as `path` + `title` + first summary line — capped like the overview; `lite` one-shot via `LLMClient.chat`; the instruction asks for a **1–3 sentence** plain-text, grounded description of what the folder's documents cover (shorter than a document summary — there can be hundreds of folders); non-empty validation, else `LLMError` → **fail-soft** (log, keep the previous summary — an old outline is better than none, the phase-31 rule).
|
||||
- **Gate:** regenerate only when the import changed the KB (the overview's change gate: added/updated > 0) or the table is empty; `--limit` debug runs skip (mirror the overview); a `lite` failure never fails the sync (log + continue — the overview's `overview=failed` status token is NOT extended: folder summaries are auxiliary, the KB is the product).
|
||||
- **Scope rules:** summarize folders with **≥ 2 documents** (a single-document folder is fully described by its one file line); after a changed sync, DELETE rows for folders that no longer have ≥ 2 documents (pruned/renamed folders — a 3→1 doc folder's summary goes stale and is dropped); rows for untouched folders persist (an unchanged folder's summary is still true).
|
||||
|
||||
### Accuracy & performance gate (task 05)
|
||||
`TOOL_CALLING_TESTING.md`'s controlled loop: `uv run python -m scripts.agent_realmodel_check --restore --mode fixture` — the 10-turn fixture battery against the live model through the real grounded path. The verdict line + per-turn wall times + the comparison against the recorded phase-70/72 baseline go into `TOOL_CALLING_TESTING.md` (dated section). Accuracy (contract/verdict) is the gate; wall time is reported. If a turn regresses, iterate on the ls copy levers (output format / `TOOLS_SECTION` teaching) with the micro-loop (`--turns 3`) per the methodology, then re-run the full battery.
|
||||
|
||||
## Tasks
|
||||
1. `01_folder_summary_model.md` — migration 0017 + `FolderSummary` model + `app/rag/folder_summaries.py` generator (marker, fail-soft, prune) + mock-LLM branch
|
||||
2. `02_sync_wiring.md` — change-gated generation in both sync paths (`scripts/import_docs.py`, `app/api/sync.py`)
|
||||
3. `03_ls_tree.md` — the three-level `ls` rewrite (`app/rag/agent.py`) + `AGENT_TOOLS` description + `TOOLS_SECTION` teaching
|
||||
4. `04_e2e_drilldown.md` — dedicated Playwright suite: scripted drill-down turns, tree shapes, cap + note
|
||||
5. `05_realmodel_battery.md` — controlled 10-turn battery vs baseline, verdict recorded in `TOOL_CALLING_TESTING.md`
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: folder extraction + grouping; summary prompt building + capped input + fail-soft + prune; generator unit (mock LLM); sync wiring integration (both paths, change-gated, fail-soft, same-transaction convention); `ls` tree output for every level/edge (empty source, unknown source, unknown folder, root vs nested, single-doc folder, 50+ file cap + note); `holder.tool_calls` accounting; `read`/`grep` untouched.
|
||||
- Coverage: **>90%** on new/modified code (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||
- E2E: `uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov` in isolation; the existing agent-tool suites (`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_harness_aligned_tools.py`, `test_search_tool.py`, `test_grep_regex_teaching.py`, `test_response_to_docs.py`) stay green in isolation.
|
||||
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `ls()` lists sources with summaries; `ls(source)` / `ls(source/folder)` drill down; file lines capped at 50 + grep pointer
|
||||
- [ ] folder summaries exist in the DB after a changed sync and are visible in `ls` output; a `lite` outage leaves old summaries intact and the sync green
|
||||
- [ ] `read`/`grep` behavior byte-identical for their contracts (existing suites green)
|
||||
- [ ] the 10-turn fixture battery verdict is recorded in `TOOL_CALLING_TESTING.md` and accuracy is at or above the phase-70/72 baseline (or an owner-acknowledged tradeoff is written down)
|
||||
- [ ] test suite green, coverage >90%, ruff + pyright clean
|
||||
- [ ] no behavior change in completed phases; one atomic Conventional Commit, `--no-gpg-sign` (e.g. `feat(agent): make ls a drill-down tree with sync-time folder summaries`)
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 01 — Folder summary storage + generator (the "smart" half)
|
||||
|
||||
**Phase:** `94_ls_tree_drilldown` · **Source:** `TODO.md:4` — "We should add a level of "smart" to this by summarizing what each folder contains (at sync time) so when the LLM calls list it gets a summary of what's in each directory."
|
||||
**Story:** n/a (owner TODO item).
|
||||
|
||||
## Objective
|
||||
Store per-folder plain-text summaries and generate them with the `lite` model — a new `folder_summaries` table, a `FOLDER_SUMMARY_MODE` one-shot generator mirroring the KB-overview contract (change-gated, fail-soft, E2E-mockable), and the prune rule for stale folders.
|
||||
|
||||
## Work
|
||||
1. `alembic/versions/0017_folder_summaries.py` (down revision `0016_ui_settings_semantic`, the head after phase 93) — new table `folder_summaries`: PK `(source, folder_path)` (String lengths mirroring `Document.source` / `Document.path` in `app/models.py`), `summary` `Text` NOT NULL, `updated_at` `timestamptz` (server default now, house style); **tested downgrade** drops it (A13).
|
||||
2. `app/models.py` — `FolderSummary` model with a docstring citing the phase-94 design (`00_phase.md`): `folder_path = ""` is the source root; rows exist only for folders with ≥ 2 documents.
|
||||
3. `app/rag/folder_summaries.py` (new module — mirror `app/rag/overview.py`'s shape and house docstring style):
|
||||
- `FOLDER_SUMMARY_MODE = "FOLDER_SUMMARY_MODE"` — system-prompt marker the deterministic E2E mock keys on (same convention as `SUMMARY_MODE` / `KB_OVERVIEW_MODE`).
|
||||
- `FOLDER_SUMMARY_INSTRUCTION` — the locked `lite` instruction: a **1–3 sentence** plain-text summary of what the folder's documents cover (natural language, no markdown, strictly grounded in the listed titles/paths/summary lines).
|
||||
- `folder_of(path) -> str` — the directory prefix before the last `/` (`""` for root-level files); module-level so unit tests can drive it.
|
||||
- `group_by_folder(rows) -> dict[(source, folder_path), list[docs]]` — module-level; ONE concept end to end (document it in the module docstring): a folder row's documents are its **recursive subtree** — every document whose path equals the folder or starts with `folder + "/"` — exactly the set the `ls` count rule in `00_phase.md` counts. Candidate rows per source: `folder_path ""` (the source root — ALL of the source's documents, this is the top-level source summary) + every distinct folder prefix of an indexed path. A doc under `a/b/` therefore contributes to BOTH the `a` row and the `a/b` row (and the `""` row) — that is intended: each level's listing shows its own accurate subtree summary.
|
||||
- `summarize_folder(source, folder_path, docs, llm) -> str` — build the prompt from the folder's documents (`path`, `title`, first summary line each), cap the input (reuse the overview's cap constant or its own — the executor picks, default ≤ 8000 chars) with the shared `TRUNCATION_MARKER` on overflow, one `LLMClient.chat` call, non-empty validation (else `LLMError` — the client already rejects empty content; re-assert defensively, the summarizer's rule).
|
||||
- `generate_folder_summaries(db, llm, *, skip: bool = False) -> dict` — the orchestrator: `skip` (the `--limit` case) is a no-op; group; for each folder with **≥ 2 documents** call `summarize_folder` and UPSERT (fail-soft per folder: catch `LLMError`, log, keep the previous row); DELETE rows whose folder no longer has ≥ 2 documents; return a small stats dict (generated/failed/pruned) for logging. Only flush — the CALLER commits (the phase-53 `bump_sources_version` convention: the sync path owns the transaction).
|
||||
4. `tests/e2e/mock_llm.py` — a canned `FOLDER_SUMMARY_MODE` branch (the mock keys on the marker in the system prompt): return a deterministic one-liner, e.g. `"Fixture folder summary for <folder path>."` (the exact template is the executor's call — it must name the folder so E2E can assert on it).
|
||||
|
||||
- ASSUMPTION: 1–3 sentence summaries (the TODO says "summarizing what each folder contains" — folders are skims, not reads; the document summary's 3–6 sentences would blow up a 20-folder listing).
|
||||
- ASSUMPTION: ≥ 2 documents per summarized folder (a single-doc folder's one file line IS its summary — no `lite` burn).
|
||||
- ASSUMPTION: fail-soft = log + keep previous (the phase-31 "an old outline is better than none" rule); a `lite` outage never fails the sync (no new status token — the KB overview's `overview=failed` token stays overview-specific).
|
||||
- ASSUMPTION: subtree (recursive) grouping — the summary scope equals the `ls` count scope at every level (one mental model for the LLM: the number next to a folder is the number of documents its summary describes); the ≥ 2 rule and the prune rule both apply to the recursive count.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (`tests/unit/`, new `test_folder_summaries.py`): `folder_of` (root, one level, deep); `group_by_folder` (multi-source, nested — a doc under `a/b/` present in the `a`, `a/b`, and `""` groups; recursive-count < 2 folders excluded); prompt building (cap + `TRUNCATION_MARKER` on overflow, ordering stable); `summarize_folder` non-empty validation; `generate_folder_summaries` with a fake `LLMClient` — happy path upsert, per-folder fail-soft (one folder raises → others land, previous kept, stats right), prune (a folder dropping below 2 docs loses its row; an untouched folder's row survives), `skip=True` no-op.
|
||||
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] migration up/down green; `FolderSummary` + generator importable and unit-tested
|
||||
- [ ] the E2E mock returns the canned folder summary when the marker is present
|
||||
- [ ] full test suite green, coverage >90%
|
||||
- [ ] no behavior change in completed work (the agent/sync suites green)
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 02 — Sync wiring: folder summaries regenerate change-gated, both paths
|
||||
|
||||
**Phase:** `94_ls_tree_drilldown` · **Source:** `TODO.md:4` — "…summarizing what each folder contains (at sync time)…" (this task is the *at sync time* part)
|
||||
**Story:** n/a (owner TODO item).
|
||||
|
||||
## Objective
|
||||
Both sync paths regenerate folder summaries in the same transaction as the KB changes, change-gated like the KB overview, best-effort/fail-soft, with `--limit` debug runs skipping — mirroring the phase-31/53 overview + version-bump conventions exactly.
|
||||
|
||||
## Work
|
||||
1. `scripts/import_docs.py` — in `_run()` next to the existing KB-overview regeneration (the change-gated block after `import_sources`): call `generate_folder_summaries(session, llm, skip=<the --limit flag>)` under the same gate the overview uses (added/updated > 0) **or** when the `folder_summaries` table is empty (first run after migration 0017 — the `_overview_row_exists` pattern, extended to a table-empty check). Same event loop, same `LLMClient` instance (the phase-31 convention). Log the stats dict (generated/failed/pruned) on the run's summary line — the line-extension house rule (PLAN §9 ample logging).
|
||||
2. `app/api/sync.py` — in the in-process sync task, at step 5 (next to the overview regeneration in the docstring's step list): the identical call + gate (`--limit` has no equivalent here; the sync button always runs the full walk). A `lite` failure in folder summaries must NOT flip the sync to failed (the overview's fail-soft is the template) and must NOT block the `bump_sources_version` (the bump stays change-gated on the KB, not on the summaries).
|
||||
3. Transaction convention: `generate_folder_summaries` only **flushes** (its docstring says so) — each sync path commits in its own transaction, exactly like `bump_sources_version` (phase 53) and the overview write, so a failed sync never half-writes summaries.
|
||||
4. Docstrings: update `scripts/import_docs.py`'s module docstring (the "refresh the stored KB overview" paragraph) and `app/api/sync.py`'s step list to name the folder-summary step.
|
||||
|
||||
- ASSUMPTION: the sync-button path (no `--limit` concept) always regenerates on a changed KB; the table-empty first-run trigger applies to both paths.
|
||||
- ASSUMPTION: no new sync-status surface (the `GET /api/sync/status` shape is untouched — folder summaries are auxiliary; failures are log-only).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration (extend the overview pattern — `tests/integration/test_import_docs_overview.py` is the template): a changed import (fake `LLMClient` keyed on `FOLDER_SUMMARY_MODE`) upserts the expected rows in the same transaction; an unchanged re-import burns **zero** `lite` calls; a folder whose subtree dropped below 2 docs is pruned; one folder raising `LLMError` keeps the previous row, lands the others, and leaves the run's exit code / sync status green; `--limit` (script path) skips generation; a fresh DB (table empty) generates on an unchanged walk.
|
||||
- The existing `tests/integration/test_sync_api.py` + `test_import_docs_git.py` stay green (status shape unchanged).
|
||||
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] after `uv run python -m scripts.import_docs` (changed KB) and after `POST /api/sync` (changed KB), `folder_summaries` holds one row per ≥ 2-doc folder subtree, visible in the run logs
|
||||
- [ ] unchanged re-syncs make zero `FOLDER_SUMMARY_MODE` calls (asserted by the fake client)
|
||||
- [ ] a `lite` outage leaves old summaries intact and both sync paths report success
|
||||
- [ ] full test suite green, coverage >90%
|
||||
- [ ] no behavior change in completed work
|
||||
@@ -0,0 +1,56 @@
|
||||
# Task 03 — The three-level `ls` rewrite (+ tool description + prompt teaching)
|
||||
|
||||
**Phase:** `94_ls_tree_drilldown` · **Source:** `TODO.md:4` — "I think list should work like a filesystem tree where it shows only the current "folder" (or at the top level, the list of synced projects) and then allow the LLM to drill down into the tree structure as it needs. … so when the LLM calls list it gets a summary of what's in each directory."
|
||||
**Story:** n/a (owner TODO item).
|
||||
|
||||
## Objective
|
||||
`ls` lists one tree level per call — sources at the top, subfolders + files below — with the stored summaries shown next to each folder, the file lines capped, and the prompt/tool-description teaching updated so the model drills instead of flooding.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — new module-level helpers (monkeypatchable, house style):
|
||||
- `ls_top(db) -> list[tuple[str, int, str | None]]` — registered sources in `list_source_names` order (registry truth, the phase-70/72 invariant — a source with 0 docs still lists), each with its recursive document count and its stored summary (`folder_summaries` row for `(source, "")` — `None` when absent).
|
||||
- `ls_folder(db, source, folder) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]` — for one folder: (a) its direct subfolders in path order, each `(sub_path_relative_to_source, recursive_count, summary_or_None)`; (b) its direct file lines `(source, path, title)` in path order (catalog order — the same order `GET /api/docs` serves); (c) the TOTAL direct-file count (pre-cap, for the note). Subfolder/file membership is pure SQL prefix logic on `Document.path` (the existence rule below).
|
||||
- Folder existence: folder `F` under registered source `S` exists ⟺ `F == ""` OR some indexed path `p` of `S` satisfies `p.startswith(F + "/")` (a document's OWN path is never a folder — nothing starts with `path + "/"`).
|
||||
2. `app/rag/agent.py` — rewrite the `ls` branch of `_execute_tool` (the `list_catalog`-based block):
|
||||
- **No path** → the top-level listing, pinned template:
|
||||
```
|
||||
{N} sources:
|
||||
|
||||
{source} — {n} documents
|
||||
{summary}
|
||||
```
|
||||
one block per source (registry order); the indented summary line only when stored; no blank line between blocks.
|
||||
- **Path = registered source** (no `/`) → its root folder, pinned template:
|
||||
```
|
||||
{identity} — {n_files} documents, {n_folders} folders:
|
||||
|
||||
{sub}/ — {m} documents: {summary}
|
||||
|
||||
source: {S} | path: {p} | title: {t}
|
||||
…and {n_files − 50} more documents in this folder — use grep (pattern) to find a specific one.
|
||||
```
|
||||
`identity` = the source name; subfolder lines 2-space-indented, path order, `: {summary}` only when stored; file lines EXACTLY the existing `source: X | path: Y | title: Z` format (the canonical `read`/`grep` identity — unchanged); the cap note only when `n_files > 50` (the cap is a pinned module constant `LS_MAX_FILE_LINES = 50`). A registered source with no documents: the header line alone (`… — 0 documents, 0 folders:`) — the old `0 documents:` behavior preserved in spirit.
|
||||
- **Path = `source/folder…`** (contains `/`) → split at the first `/`; unknown source → the existing `NO_SOURCE_NOT_A_DIRECTORY` refusal (teaching parenthetical intact); unknown folder → the NEW `NOT_A_FOLDER` refusal line, phase-72 teaching style (one line, argument echoed, parent's subfolders listed so the model self-corrects next round, e.g. `'homelab/netwoking' is not a folder — homelab has: backups/ containers/ networking/`); otherwise the same template as the root level with `identity = source + "/" + folder`.
|
||||
- The old `LS_PATH_NOT_A_SOURCE` refusal (a `/` in the scope was always wrong) is **deleted** — a `/` now names a folder; update the unit tests that pin it (see Testing).
|
||||
- `holder.tool_calls += 1` on every successful listing (top/root/folder); refusals count in nothing (unchanged).
|
||||
3. `app/rag/agent.py` — `AGENT_TOOLS` `ls` entry: rewrite the function description to the tree contract (pinned copy):
|
||||
> "List the knowledge base as a tree, one level at a time. With no path: the synced sources — each with its document count and a summary of its contents. With a source name (no '/'): that source's top-level folders and files. With a `source/folder` path: that folder's subfolders and files. Folder lines carry a summary of what the folder contains. File lines are `source: X | path: Y | title: Z` — use the combined `source/path` with `read` and `grep`. Call one tool at a time — wait for this result before your next call."
|
||||
and the `path` parameter description: "Optional — a source name (e.g. 'homelab') to list its top level, or a `source/folder` path to drill down (e.g. 'homelab/active'). Omit it to list every source."
|
||||
4. `app/rag/prompts.py` — `TOOLS_SECTION`: rewrite the `ls` teaching paragraph to the drill-down contract (top level = sources; folders list subfolders + files only — never the whole KB in one call; summaries tell what's in a folder before drilling; `grep` stays the locator for finding one document without listing). `read`/`grep` teaching untouched.
|
||||
5. `app/rag/agent.py` module docstring — the phase-70 tool-surface paragraph: add the phase-94 revision note (owner permission 2026-09-10, `TODO.md` L4 — the `ls` RESULT format + `path` semantics changed; the tool NAME and the `read`/`grep` contract are untouched; PLAN.md is being redone by the owner, the decision is recorded in `00_phase.md`).
|
||||
|
||||
- ASSUMPTION: `LS_MAX_FILE_LINES = 50` pinned module constant (no env var — the TODO asks for a shape change, not a knob; the constant lives next to `SEARCH_MAX_MATCHES`).
|
||||
- ASSUMPTION: the exact punctuation of the three pinned templates above is the contract (unit tests pin it byte-for-byte, the house pattern); "documents"/"folders" stay unpluralized (stable for the model, like today's `0 documents:` line).
|
||||
- ASSUMPTION: the NOT-A-FOLDER refusal lists the PARENT folder's direct subfolders (bounded — the parent's own listing, so no new flood path).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (`tests/unit/test_agent.py` — extend; it pins the tool surface and refusal copy today): `ls_top` (registry order, 0-doc source, summary present/absent); `ls_folder` (subfolder recursion + counts, direct-file membership, path order, the file-path-is-not-a-folder rule); the three result templates byte-for-byte (top / root / nested, empty folder, summary absent); the 50-line cap + note (51 files → 50 lines + `…and 1 more…`); refusals (unknown source → `NO_SOURCE_NOT_A_DIRECTORY` prefix intact; unknown folder → `NOT_A_FOLDER` with the parent's subfolders; the deleted `LS_PATH_NOT_A_SOURCE` is gone); `holder.tool_calls` accounting (success vs refusal); `read`/`grep` branches byte-identical for their contracts.
|
||||
- Unit: `AGENT_TOOLS` `ls` description + `path` parameter + `TOOLS_SECTION` contain the tree contract (the house string-pin tests updated, not deleted).
|
||||
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `ls()` / `ls(source)` / `ls(source/folder)` render the three pinned templates against a multi-folder DB (unit-pinned)
|
||||
- [ ] a 500-file folder costs the model 50 file lines + one note, never 500
|
||||
- [ ] the `read`/`grep` contracts and their suites are byte-identical (no change)
|
||||
- [ ] full test suite green, coverage >90%
|
||||
- [ ] no behavior change in completed work (deflected turns, the round cap, the log line all untouched)
|
||||
@@ -0,0 +1,33 @@
|
||||
# Task 04 — E2E: the scripted drill-down (the dedicated story suite)
|
||||
|
||||
**Phase:** `94_ls_tree_drilldown` · **Source:** `TODO.md:4` — the whole item (this task is the story gate: the LLM drills source → folder → file through the UI, summaries visible, cap held)
|
||||
**Story:** n/a (owner TODO item — one Playwright file per story, run in isolation, A16).
|
||||
|
||||
## Objective
|
||||
`tests/e2e/test_ls_tree_drilldown.py` proves the new architecture end to end: a folder-structured KB synced under the deterministic mock LLM, scripted agent turns that drill `ls()` → `ls(source)` → `ls(source/folder)` → `read(source/file)`, the drill-down tool lines rendering in the UI, the folder summary the mock generated appearing in the model's context, and the 50-line cap holding on a wide folder.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_ls_tree_drilldown.py` (new, isolated — `tests/e2e/conftest.py` fixtures `app_server` + `mock_llm`; admin login via `tests/e2e/auth_helpers.py`):
|
||||
- **Seed:** a temp local directory tree — two sources (two temp roots), each with nested folders, e.g. `alpha/{one,two}/…md` and `beta/{gamma}/…md` (the `tests/e2e/test_local_directory_sources.py` pattern for registering a local source + Sync); the mock's canned `FOLDER_SUMMARY_MODE` branch (task 01) makes sync store a deterministic summary per ≥ 2-doc folder, so the test can assert on exact summary text.
|
||||
- **Scripted turns** (extend `tests/e2e/mock_llm.py`'s scripted-tool-turn machinery — the `tests/e2e/test_agent_document_tools.py` pattern — with a drill-down script):
|
||||
1. "What sources are indexed?" → the mock calls `ls()` → assert the `.tool-call` "🔎 Listing documents" line renders, and the mock's grounded answer quotes the top-level shape (a `— N documents` line per source; the canned source-root summary text for a source with ≥ 2 docs — the mock echoes what it received, the house scripted-turn way of asserting on tool results).
|
||||
2. "What's in source `alpha`?" → the mock calls `ls("alpha")` → assert the folder lines (`one/ — …`, `two/ — …` + their summaries) and any root file lines, via the mock's echo.
|
||||
3. "List `alpha/two`" → the mock calls `ls("alpha/two")` → assert the file lines in the exact `source: X | path: Y | title: Z` format.
|
||||
4. "Read the file" → the mock calls `read("alpha/two/<file>.md")` → the "📄 Reading …" line renders and the final answer cites the document (grounded-turn contract, the phase-37 assertion pattern).
|
||||
- **Cap:** one source with a folder holding 51 tiny files (51 one-line `.md` files in the temp tree) → the mock's `ls` of that folder echoes exactly 50 file lines + the `…and 1 more documents…` note.
|
||||
- **Refusal visible to the model:** one turn where the mock calls `ls("alpha/nope")` → the next round the mock (scripted) recovers with `ls("alpha")` — the phase-72 self-correction contract now carries the tree's NOT-A-FOLDER teaching.
|
||||
2. Run in isolation: `uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov` (db up: `podman compose up -d db`).
|
||||
3. Regression gate, in isolation: `test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_harness_aligned_tools.py`, `test_search_tool.py`, `test_grep_regex_teaching.py`, `test_sync_button.py`, `test_local_directory_sources.py`.
|
||||
|
||||
- ASSUMPTION: assertions on tool RESULTS go through the deterministic mock echoing the received result into its final answer (the mock is the only lens the E2E has on the LLM's context — the house pattern from the agent-tool suites); the DOM assertions cover the tool lines + the answer.
|
||||
- ASSUMPTION: the drill-down mock script is a NEW script in `mock_llm.py` (keyed like the existing ones), not a change to the existing scripts — the old suites stay green unmodified.
|
||||
|
||||
## Testing & Quality
|
||||
- This IS the E2E task; additionally the mock's new branch/script needs a unit-level smoke (the mock is a real OpenAI-compatible server — the conftest boots it; no extra unit needed beyond the E2E itself, per the existing agent suites' practice).
|
||||
- Coverage: **>90%** on `app/` (this task adds test code only — keep the gate green).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the suite is green in isolation and encodes the full drill-down: sources → folders (with summaries) → files → grounded read
|
||||
- [ ] the 51-file folder costs the model 50 lines + the note (asserted via the mock echo)
|
||||
- [ ] the NOT-A-FOLDER teaching is visible to the model and a scripted recovery works
|
||||
- [ ] full test suite green, coverage >90%
|
||||
@@ -0,0 +1,33 @@
|
||||
# Task 05 — Accuracy & performance: the controlled tool-calling battery vs baseline
|
||||
|
||||
**Phase:** `94_ls_tree_drilldown` · **Source:** `TODO.md:4` — "Run tests against this new architecture and ensure accuracy and performance aren't too badly impacted."
|
||||
**Story:** n/a (owner TODO item).
|
||||
|
||||
## Objective
|
||||
Run the controlled 10-question fixture battery (the `TOOL_CALLING_TESTING.md` methodology — real configured chat model, restored fixture KB, the exact mirror of the grounded path) against the new `ls` architecture, iterate on the copy levers if any turn regresses, and record the verdict + timings + baseline comparison in `TOOL_CALLING_TESTING.md`.
|
||||
|
||||
## Work
|
||||
1. Fixture snapshot freshness: if `tests/fixtures/test_kb.dump.sql` predates migration 0017 (it does — it has no `folder_summaries` rows), refresh the snapshot: restore, re-import the fixture directory through the real pipeline (real embeddings — the methodology's ~1–2 s rebuild), let the sync-time folder summaries generate against the live `lite` endpoint, and re-dump (one transaction snapshot, the house "hand-written, unguessable, fixed-size KB snapshotted to a SQL dump" rule). The dump's `sources_meta`/`kb_overview` rows follow the same refresh.
|
||||
2. The loop:
|
||||
```bash
|
||||
podman compose up -d db
|
||||
uv run python -m scripts.agent_realmodel_check --restore --mode fixture
|
||||
```
|
||||
(full 10 turns; the micro-loop `--turns 3` is the iteration tool when a variant looks off).
|
||||
3. Review the 10 questions' "unambiguously correct tool behavior" against the tree: where the tree changes the correct answer (e.g. a "list everything" expectation becomes "drill into the right folder"), update the battery question's expected-behavior note in `TOOL_CALLING_TESTING.md` — the question itself stays; the correct-behavior definition moves with the owner-permitted surface change. If a question's intent no longer maps, flag it in the recorded section (do NOT silently rewrite a question).
|
||||
4. Iterate only on copy levers (the `ls` output templates / `AGENT_TOOLS` description / `TOOLS_SECTION`) if a turn regresses — micro-loop first (`--turns 3`), full battery for the verdict. Any template change here re-pins the task-03 unit tests in the same commit.
|
||||
5. Record a dated section in `TOOL_CALLING_TESTING.md` (the house format — per-turn lines with wall seconds + the verdict line + total wall): the run against the new architecture, the per-turn verdicts, the comparison against the last recorded baseline (phase 70/72 numbers in the file), and the conclusion. Accuracy (contract/verdict) is the gate per the TODO; wall time is reported as "performance."
|
||||
|
||||
- ASSUMPTION: "not too badly impacted" = **accuracy at or above baseline** (no contract/verdict regression on the 10-question battery) and **wall time within ~20% of the baseline total** (the drill-down can add rounds — the tree is a few extra `ls` hops; a modest wall increase is expected and reportable, a big one is not). A violation ⇒ iterate the copy levers; if still in violation, STOP and write the tradeoff into the recorded section for the owner (the TODO leaves the threshold to judgment — the executor must not silently accept).
|
||||
- ASSUMPTION: the battery runs against the LIVE aipi endpoint (the methodology's premise — `E2E_REAL_LLM` is the E2E opt-in; `agent_realmodel_check` is the real-model tool by definition). If the endpoint is unreachable, the task is blocked — report it, don't fake the verdict.
|
||||
|
||||
## Testing & Quality
|
||||
- No new app code is expected from this task (docs + fixture dump + possibly the task-03 copy levers, re-pinned). The full suite + coverage >90% stays green; ruff + pyright clean.
|
||||
- The recorded section in `TOOL_CALLING_TESTING.md` is the artifact — it must be reproducible from the file alone (commands, timings, verdicts, baseline diff).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `TOOL_CALLING_TESTING.md` carries the dated phase-94 section: 10/10 per-turn lines, verdict, total wall, baseline comparison, conclusion
|
||||
- [ ] accuracy ≥ baseline (or the recorded owner-acknowledgment exists)
|
||||
- [ ] wall time within the 20% band (or the recorded tradeoff exists)
|
||||
- [ ] fixture dump refreshed (carries `folder_summaries` rows) and `--restore` still ~0.03 s hot
|
||||
- [ ] full test suite green, coverage >90%
|
||||
Reference in New Issue
Block a user