phase: 94_ls_tree_drilldown
All green. Verification complete. **Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)** - Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal - Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths - Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met - `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %) - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched) - Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed - Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol) **Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
# Phase 94 — `ls` becomes a drill-down tree with sync-time folder summaries
|
||||
|
||||
**Source:** `TODO.md` L4 — "Brain of reese can end up indexing thousands of files. When the llm calls `list` it can end up filling the context entirely with just the document lists. I think list should work like a filesystem tree where it shows only the current "folder" (or at the top level, the list of synced projects) and then allow the LLM to drill down into the tree structure as it needs. We should add a level of "smart" to this by summarizing what each folder contains (at sync time) so when the LLM calls list it gets a summary of what's in each directory. Run tests against this new architecture and ensure accuracy and performance aren't too badly impacted."
|
||||
**Story:** n/a (owner TODO item — agent tooling on `37_agent_document_tools` / `45_agent_unlimited_tools` / `68_search_tool` / `70_harness_aligned_tools`).
|
||||
**Context:** Today `ls` (the harness-aligned surface, phase 70 — `app/rag/agent.py` `AGENT_TOOLS`) lists **every** indexed document as `source: X | path: Y | title: Z` lines (or one source's documents when `path` is a source name) — with a 1,340-document KB that is the entire context in one call. `read`/`grep` take the combined `source/path` identity; `grep` is the whole-KB locator. The one-shot `lite` machinery exists: document summaries (`app/rag/summarizer.py`, `SUMMARY_MODE`) and the KB overview (`app/rag/overview.py`, `KB_OVERVIEW_MODE`) — both change-gated at import, best-effort/fail-soft, with a marker the deterministic E2E mock (`tests/e2e/mock_llm.py`) keys on. Sync paths: `scripts/import_docs.py` (`_run()`, the overview hook) and `POST /api/sync` (`app/api/sync.py`, step 5). The controlled accuracy/performance methodology is `TOOL_CALLING_TESTING.md` + `uv run python -m scripts.agent_realmodel_check --restore --mode fixture` (the 8-document, 2-source, **folder-structured** fixture KB in `tests/fixtures/test_kb.dump.sql`: `deployments/{ansible,ci,quadlet}` + `homelab/{backups,containers,networking}`).
|
||||
|
||||
## Objective
|
||||
Rebuild `ls` as a filesystem-style tree the LLM drills through: top level lists the synced projects (sources) with a per-source summary; `ls` of a source lists its folders (each with a sync-time summary) + top-level files; `ls` of a folder lists its subfolders + files. Folder summaries are generated at sync time by the `lite` model (change-gated, fail-soft). The controlled tool-calling battery then proves accuracy and performance are not badly impacted.
|
||||
|
||||
## Dependencies
|
||||
- `93_theme_semantic_completion` (todo) — queue order only; no code dependency (different subsystems).
|
||||
|
||||
## Owner-permitted decision recorded here (PLAN.md is being redone by the owner)
|
||||
- **Tool-surface revision (owner permission 2026-09-10, TODO.md L4):** the `ls` RESULT format and `path` semantics change (tree drill-down); the `ls` NAME and the `read`/`grep` contract (combined `source/path`) are untouched. This extends the phase-70 harness-aligned surface per the owner's explicit request; the prompt teaching (`TOOLS_SECTION`) is rewritten to match in the same change.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
### Tree shape (task 03)
|
||||
Document paths are already tree-like (`deployments/ansible/lab-inventory.md`); a **folder** is a path prefix. Three `ls` levels, one argument:
|
||||
|
||||
| Call | Result |
|
||||
|---|---|
|
||||
| `ls()` (no path) | every registered source (registry order, `list_source_names`): `name — N documents` + an indented summary line when one is stored (folders with no docs still list as `0 documents`) |
|
||||
| `ls(path=<source>)` | the source's **root folder**: each subfolder ` <folder>/ — N documents[: summary]` (count = documents with `path == folder` or `path startswith folder + "/"`), then the root's own file lines `source: X | path: Y | title: Z` |
|
||||
| `ls(path=<source>/<folder>[/…])` | that folder's subfolders (same shape, paths relative to the source) + the folder's own file lines |
|
||||
|
||||
- **File-line cap:** a folder's own files list at most **50** lines (path order — catalog order), then one deterministic note: `…and {N−50} more documents in this folder — use grep (pattern) to find a specific one.` (folds the old whole-KB flood into one bounded level; `grep` stays the locator — no new tool).
|
||||
- **Summaries:** shown for the SOURCE (top level) and for each SUBFOLDER, when stored; absent → the count line only (no placeholder).
|
||||
- **Refusals (adapted from the phase-70/72 teaching lines, same style — one line, argument echoed, counts in nothing, consumes a round):** unknown first segment → the existing no-source refusal; a `source/…` argument whose folder matches no indexed prefix → a NOT-A-FOLDER line teaching the drill-down (echo the argument, list the parent folder's subfolders so the model can self-correct in the next round). `ls` of a registered source with no documents keeps the `0 documents:` behavior.
|
||||
- **`holder.tool_calls`** increments on every successful listing (top/root/folder); refusals count in nothing (unchanged house rule).
|
||||
|
||||
### Folder summaries (tasks 01, 02)
|
||||
- **Storage:** new table `folder_summaries` — PK `(source, folder_path)` (column types mirror `Document.source` / `Document.path`), `summary` `Text`, `updated_at` `timestamptz`. `folder_path = ""` is the SOURCE ROOT (the top-level source summary); a folder is any other path prefix.
|
||||
- **Generation** (`app/rag/folder_summaries.py`, mirroring `app/rag/overview.py`): marker `FOLDER_SUMMARY_MODE` (the E2E mock keys on it — `tests/e2e/mock_llm.py` gains the canned branch); input per folder = its **recursive subtree** — every document whose path equals the folder or starts with `folder + "/"` (the same set the `ls` count shows; `folder_path ""` = the whole source), each as `path` + `title` + first summary line — capped like the overview; `lite` one-shot via `LLMClient.chat`; the instruction asks for a **1–3 sentence** plain-text, grounded description of what the folder's documents cover (shorter than a document summary — there can be hundreds of folders); non-empty validation, else `LLMError` → **fail-soft** (log, keep the previous summary — an old outline is better than none, the phase-31 rule).
|
||||
- **Gate:** regenerate only when the import changed the KB (the overview's change gate: added/updated > 0) or the table is empty; `--limit` debug runs skip (mirror the overview); a `lite` failure never fails the sync (log + continue — the overview's `overview=failed` status token is NOT extended: folder summaries are auxiliary, the KB is the product).
|
||||
- **Scope rules:** summarize folders with **≥ 2 documents** (a single-document folder is fully described by its one file line); after a changed sync, DELETE rows for folders that no longer have ≥ 2 documents (pruned/renamed folders — a 3→1 doc folder's summary goes stale and is dropped); rows for untouched folders persist (an unchanged folder's summary is still true).
|
||||
|
||||
### Accuracy & performance gate (task 05)
|
||||
`TOOL_CALLING_TESTING.md`'s controlled loop: `uv run python -m scripts.agent_realmodel_check --restore --mode fixture` — the 10-turn fixture battery against the live model through the real grounded path. The verdict line + per-turn wall times + the comparison against the recorded phase-70/72 baseline go into `TOOL_CALLING_TESTING.md` (dated section). Accuracy (contract/verdict) is the gate; wall time is reported. If a turn regresses, iterate on the ls copy levers (output format / `TOOLS_SECTION` teaching) with the micro-loop (`--turns 3`) per the methodology, then re-run the full battery.
|
||||
|
||||
## Tasks
|
||||
1. `01_folder_summary_model.md` — migration 0017 + `FolderSummary` model + `app/rag/folder_summaries.py` generator (marker, fail-soft, prune) + mock-LLM branch
|
||||
2. `02_sync_wiring.md` — change-gated generation in both sync paths (`scripts/import_docs.py`, `app/api/sync.py`)
|
||||
3. `03_ls_tree.md` — the three-level `ls` rewrite (`app/rag/agent.py`) + `AGENT_TOOLS` description + `TOOLS_SECTION` teaching
|
||||
4. `04_e2e_drilldown.md` — dedicated Playwright suite: scripted drill-down turns, tree shapes, cap + note
|
||||
5. `05_realmodel_battery.md` — controlled 10-turn battery vs baseline, verdict recorded in `TOOL_CALLING_TESTING.md`
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: folder extraction + grouping; summary prompt building + capped input + fail-soft + prune; generator unit (mock LLM); sync wiring integration (both paths, change-gated, fail-soft, same-transaction convention); `ls` tree output for every level/edge (empty source, unknown source, unknown folder, root vs nested, single-doc folder, 50+ file cap + note); `holder.tool_calls` accounting; `read`/`grep` untouched.
|
||||
- Coverage: **>90%** on new/modified code (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||
- E2E: `uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov` in isolation; the existing agent-tool suites (`test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_harness_aligned_tools.py`, `test_search_tool.py`, `test_grep_regex_teaching.py`, `test_response_to_docs.py`) stay green in isolation.
|
||||
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `ls()` lists sources with summaries; `ls(source)` / `ls(source/folder)` drill down; file lines capped at 50 + grep pointer
|
||||
- [ ] folder summaries exist in the DB after a changed sync and are visible in `ls` output; a `lite` outage leaves old summaries intact and the sync green
|
||||
- [ ] `read`/`grep` behavior byte-identical for their contracts (existing suites green)
|
||||
- [ ] the 10-turn fixture battery verdict is recorded in `TOOL_CALLING_TESTING.md` and accuracy is at or above the phase-70/72 baseline (or an owner-acknowledged tradeoff is written down)
|
||||
- [ ] test suite green, coverage >90%, ruff + pyright clean
|
||||
- [ ] no behavior change in completed phases; one atomic Conventional Commit, `--no-gpg-sign` (e.g. `feat(agent): make ls a drill-down tree with sync-time folder summaries`)
|
||||
Reference in New Issue
Block a user