refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills): phases/, user_stories/, reports/, screenshots/, validate.sh, and phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves history; runtime artifacts move alongside). Updates every reference in AGENTS.md, README.md, .gitignore, app docstrings, and test story headers. Historical KB content in data/ and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
@@ -0,0 +1,54 @@
|
||||
# Phase 63 — Unambiguous Document Listing for Agent Tools
|
||||
|
||||
**Source:** Live troubleshooting report (2026-09-01) — the `brain-of-reese` agent at `https://brain.experimental.reeseapps.com/` cannot read documents because the `list_documents` tool output uses an ambiguous `source/path — title` format that the LLM cannot reliably parse when paths contain `/` characters.
|
||||
|
||||
**Story:** `.agents/user_stories/agent-document-tools.md` (phase 37, extended)
|
||||
|
||||
**Context:** The agent's grounded-turn loop uses two tools: `list_documents` (returns the catalog) and `read_document` (reads one document). The catalog output is formatted as `source/path — title` (one line per document), but when a file lives inside a subdirectory of its source, the output looks like `brain-of-reese-main/homelab/aws-route53.md — aws-route53`. The LLM cannot reliably determine where `source` ends and `path` begins because both contain `/` separators. The thinking trace shows the LLM cycling through multiple failed attempts, each time guessing a different split.
|
||||
|
||||
The fix changes the catalog format to unambiguous `source: X | path: Y | title: Z` lines and updates the tool descriptions to tell the LLM how to parse them.
|
||||
|
||||
## Objective
|
||||
|
||||
The agent can always extract the correct `(source, path)` pair from the `list_documents` output, regardless of how many `/` characters the path contains.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `62_ui_customization` (todo, preceding — functional: no code changes in this phase touch UI or config)
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_agent_list_format.md` — change `list_catalog` output from `source/path — title` to `source: X | path: Y | title: Z`; update the `read_document` tool descriptions to reference the new format.
|
||||
2. `02_mock_and_unit_tests.md` — update `tests/e2e/mock_llm.py::_catalog_docs` to parse the new format; update `tests/unit/test_agent.py` (the "No document at" refusal line is unchanged — it names the lookup failure, not the list format).
|
||||
3. `03_e2e_document_tools.md` — validate that `tests/e2e/test_agent_document_tools.py` and `tests/e2e/test_agent_unlimited_tools.py` still pass (they assert on SSE frames and UI rendering, not on the catalog text format — the tool call `argument` field `source/path` is unchanged).
|
||||
4. `04_docs.md` — update any inline documentation that references the old format.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/unit/test_agent.py` — the "No document at" refusal message is unchanged (it reports the lookup failure, not the list format). The `_catalog_docs` mock parsing must produce identical `(source, path)` tuples for all existing test catalogs.
|
||||
- Integration: `tests/integration/test_agent_tools.py` — `list_catalog` ordering assertions are unchanged (the function returns `[(source, path, title), ...]` tuples; only the string formatting in `_execute_tool` changes).
|
||||
- E2E (mandatory, house rule): `tests/e2e/test_agent_document_tools.py` and `tests/e2e/test_agent_unlimited_tools.py` run in isolation — the SSE `argument` field (`source/path`) is unchanged; the mock's `_catalog_docs` parsing produces identical results; the tool lines and source chips render identically.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `list_catalog` output uses `source: X | path: Y | title: Z` format — verifiable by a unit test that patches `list_catalog` and checks the formatted string.
|
||||
- [ ] The `read_document` tool descriptions tell the LLM to extract `source` and `path` from the `source:` / `path:` labels.
|
||||
- [ ] `tests/e2e/mock_llm.py::_catalog_docs` parses the new format and produces identical `(source, path)` tuples for all existing test catalogs.
|
||||
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` green in isolation (DB up).
|
||||
- [ ] Regression E2E suites green in isolation: `test_chat_api.py`, `test_sse_events.py`, `test_whole_document_context.py`.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` (`.agents/` stays untracked).
|
||||
|
||||
## Locked decisions
|
||||
|
||||
- **Owner-locked (2026-09-01, troubleshooting report, A1):** the new format uses `source: X | path: Y | title: Z` — one line per document, pipe-delimited fields, no ambiguity even with deeply nested paths.
|
||||
- **Owner-locked (2026-09-01, troubleshooting report, A2):** the `read_document` tool descriptions are updated to reference the new format (tell the LLM how to parse `source:` and `path:` labels).
|
||||
- **Owner-locked (2026-09-01, troubleshooting report, A3):** the SSE `argument` field (`source/path`) is unchanged — it is the tool call argument, not the list output. The frontend tool-line rendering is unchanged.
|
||||
- **Owner-locked (2026-09-01, troubleshooting report, A4):** the "No document at …" refusal message is unchanged — it reports the lookup failure, not the list format.
|
||||
|
||||
## Commit
|
||||
|
||||
```bash
|
||||
git add app/ tests/ && git commit --no-gpg-sign -m "fix(agent): unambiguous document listing format for LLM parsing"
|
||||
```
|
||||
@@ -0,0 +1,75 @@
|
||||
# Task 01 — Change `list_catalog` output format + update tool descriptions
|
||||
|
||||
**Goal:** Make the `list_documents` catalog output unambiguous for LLM parsing, and update the `read_document` tool descriptions to tell the LLM how to extract `source` and `path`.
|
||||
|
||||
## Changes
|
||||
|
||||
### `app/rag/agent.py`
|
||||
|
||||
#### 1. Change `list_catalog` output format
|
||||
|
||||
In `_execute_tool`, the `list_documents` handler formats the catalog as:
|
||||
|
||||
```python
|
||||
# BEFORE (ambiguous when path contains /)
|
||||
listing = f"{len(rows)} documents:\n" + "\n".join(
|
||||
f"{source}/{path} — {title}" for source, path, title in rows
|
||||
)
|
||||
```
|
||||
|
||||
Change to:
|
||||
|
||||
```python
|
||||
# AFTER (unambiguous — each field is labeled)
|
||||
listing = f"{len(rows)} documents:\n" + "\n".join(
|
||||
f"source: {source} | path: {path} | title: {title}"
|
||||
for source, path, title in rows
|
||||
)
|
||||
```
|
||||
|
||||
#### 2. Update `read_document` tool descriptions
|
||||
|
||||
In `AGENT_TOOLS`, update the `source` and `path` parameter descriptions to reference the new format:
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
"source": {
|
||||
"description": (
|
||||
"The document's source (a directory basename, e.g. 'Homelab')."
|
||||
),
|
||||
},
|
||||
"path": {
|
||||
"description": (
|
||||
"The document's path relative to its source directory."
|
||||
),
|
||||
},
|
||||
|
||||
# AFTER
|
||||
"source": {
|
||||
"description": (
|
||||
"The document's source, as shown after 'source: ' in the "
|
||||
"list_documents output (e.g. 'Homelab' from "
|
||||
"'source: Homelab | path: homelab/aws-route53.md')."
|
||||
),
|
||||
},
|
||||
"path": {
|
||||
"description": (
|
||||
"The document's path, as shown after 'path: ' in the "
|
||||
"list_documents output (e.g. 'homelab/aws-route53.md' from "
|
||||
"'source: Homelab | path: homelab/aws-route53.md')."
|
||||
),
|
||||
},
|
||||
```
|
||||
|
||||
### `app/rag/prompts.py`
|
||||
|
||||
No changes needed — the prompt text references the tool names but not the output format.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `app/rag/agent.py` — `list_catalog` formatting + `AGENT_TOOLS` descriptions
|
||||
|
||||
## Verification
|
||||
|
||||
- `uv run pytest tests/unit/test_agent.py -v --no-cov` — the tool shape tests (`test_agent_tools_names_and_parameters`) check the description strings, so they must be updated to match.
|
||||
- `uv run ruff check app/rag/agent.py && uv run pyright app/rag/agent.py` — lint + types clean.
|
||||
@@ -0,0 +1,85 @@
|
||||
# Task 02 — Update mock parsing + unit tests for the new format
|
||||
|
||||
**Goal:** Update `tests/e2e/mock_llm.py::_catalog_docs` to parse the new `source: X | path: Y | title: Z` format, and update `tests/unit/test_agent.py` to match the new tool descriptions.
|
||||
|
||||
## Changes
|
||||
|
||||
### `tests/e2e/mock_llm.py` — `_catalog_docs` function
|
||||
|
||||
The current parsing uses `head.rpartition("/")` on the `source/path` head:
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||
docs: list[tuple[str, str]] = []
|
||||
for m in _messages(body):
|
||||
if m.get("role") != "tool":
|
||||
continue
|
||||
content = str(m.get("content") or "")
|
||||
if content.startswith(_READ_RESULT_PREFIX):
|
||||
continue
|
||||
for line in content.splitlines():
|
||||
head = line.split(" — ", 1)[0].strip()
|
||||
if "/" in head:
|
||||
source, _, path = head.rpartition("/")
|
||||
if source and path:
|
||||
docs.append((source, path))
|
||||
return docs
|
||||
```
|
||||
|
||||
Change to parse the new labeled format:
|
||||
|
||||
```python
|
||||
# AFTER
|
||||
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||
docs: list[tuple[str, str]] = []
|
||||
for m in _messages(body):
|
||||
if m.get("role") != "tool":
|
||||
continue
|
||||
content = str(m.get("content") or "")
|
||||
if content.startswith(_READ_RESULT_PREFIX):
|
||||
continue
|
||||
for line in content.splitlines():
|
||||
# New format: "source: X | path: Y | title: Z"
|
||||
if not line.startswith("source: "):
|
||||
continue
|
||||
parts = line.split(" | ", 2)
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
source = parts[0].removeprefix("source: ").strip()
|
||||
path_part = parts[1]
|
||||
if not path_part.startswith("path: "):
|
||||
continue
|
||||
path = path_part.removeprefix("path: ").strip()
|
||||
if source and path:
|
||||
docs.append((source, path))
|
||||
return docs
|
||||
```
|
||||
|
||||
### `tests/unit/test_agent.py` — tool description assertions
|
||||
|
||||
The test `test_agent_tools_names_and_parameters` asserts the exact description strings:
|
||||
|
||||
```python
|
||||
# BEFORE (line ~109)
|
||||
assert by_name["read_document"]["function"]["description"] == (
|
||||
"Add the full content of one more indexed document to your context"
|
||||
)
|
||||
```
|
||||
|
||||
This assertion is about the function description (not parameter descriptions), so it stays the same. But the test file may have inline assertions about the `source` and `path` parameter descriptions — check and update if any exist.
|
||||
|
||||
### `tests/unit/test_agent.py` — refusal message
|
||||
|
||||
The refusal message `"No document at S/ghost.md — check the list_documents output."` is unchanged — it reports the lookup failure, not the list format. No changes needed here.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tests/e2e/mock_llm.py` — `_catalog_docs` parsing
|
||||
- `tests/unit/test_agent.py` — only if inline assertions reference the old format (verify)
|
||||
|
||||
## Verification
|
||||
|
||||
- `uv run pytest tests/unit/test_agent.py -v --no-cov` — all agent unit tests pass.
|
||||
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` — the mock's `_catalog_docs` must produce identical `(source, path)` tuples for the test catalogs.
|
||||
- `uv run ruff check tests/ && uv run pyright tests/` — lint + types clean.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Task 03 — E2E document tools validation
|
||||
|
||||
**Goal:** Run the E2E suites for agent document tools and verify they still pass with the new catalog format.
|
||||
|
||||
## What to verify
|
||||
|
||||
The E2E tests assert on:
|
||||
1. **SSE frames** — the `argument` field of `tool` events is `source/path` (e.g., `"Deployments/example-record-file.json"`). This is the tool call argument, NOT the list output format. **Unchanged.**
|
||||
2. **UI rendering** — tool lines show "Listing documents" and "Reading <source/path>". **Unchanged.**
|
||||
3. **Source chips** — display `${s.source}/${s.path}`. **Unchanged.**
|
||||
4. **Answer content** — the mock's deterministic answer quoting the read document. **Unchanged** (the mock reads the same `(source, path)` tuples from the catalog).
|
||||
5. **Query log** — sources field stores `"source/path, source/path"`. **Unchanged.**
|
||||
|
||||
The only thing that changes is the **text of the catalog** that the mock parses. The mock's `_catalog_docs` function must produce identical `(source, path)` tuples for the test catalogs, which means the tool flow classification (`_tool_flow`) and the read step will work identically.
|
||||
|
||||
## Run commands
|
||||
|
||||
```bash
|
||||
# DB must be up: podman compose up -d db
|
||||
uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov
|
||||
uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov
|
||||
```
|
||||
|
||||
## Regression suites to run
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_chat_api.py -v --no-cov
|
||||
uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov
|
||||
uv run pytest tests/e2e/test_sse_events.py -v --no-cov
|
||||
```
|
||||
|
||||
## Expected outcome
|
||||
|
||||
All tests pass without modification (aside from the mock parsing change in task 02). The catalog text format is an internal detail of the agent loop — the tool call arguments, SSE frames, UI rendering, and answer content are all unchanged.
|
||||
|
||||
## If tests fail
|
||||
|
||||
1. Check that `_catalog_docs` produces the same `(source, path)` tuples as before for the test catalogs.
|
||||
2. Verify the mock's `_tool_flow` classification still works (it calls `_catalog_docs`).
|
||||
3. Check that the SSE `argument` field is still `source/path` (it should be — `_execute_tool` builds it from `call.arguments["source"] + "/" + call.arguments["path"]`, which is unchanged).
|
||||
|
||||
## Files changed
|
||||
|
||||
- None (verification only — all changes were in tasks 01 and 02)
|
||||
@@ -0,0 +1,44 @@
|
||||
# Task 04 — Update documentation
|
||||
|
||||
**Goal:** Update all documentation that references the old `source/path — title` catalog format to the new `source: X | path: Y | title: Z` format.
|
||||
|
||||
## Files to update
|
||||
|
||||
### `README.md` — Agent document tools section (~L168)
|
||||
|
||||
```markdown
|
||||
# BEFORE
|
||||
* **`list_documents`** — lists every indexed document, one
|
||||
`source/path — title` line each (the same order as the Sources page);
|
||||
|
||||
# AFTER
|
||||
* **`list_documents`** — lists every indexed document, one
|
||||
`source: X | path: Y | title: Z` line each (the same order as the
|
||||
Sources page);
|
||||
```
|
||||
|
||||
### `.agents/user_stories/agent-document-tools.md` — Acceptance criteria (~L38)
|
||||
|
||||
```markdown
|
||||
# BEFORE
|
||||
3. `app/rag/agent.py`: the loop — budgets from
|
||||
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
|
||||
`list_documents` tool returns the DB catalog
|
||||
(`source/path — title` lines, `/api/docs` order);
|
||||
|
||||
# AFTER
|
||||
3. `app/rag/agent.py`: the loop — budgets from
|
||||
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
|
||||
`list_documents` tool returns the DB catalog
|
||||
(`source: X | path: Y | title: Z` lines, `/api/docs` order);
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
- `grep -n "source/path — title\|source/path - title" README.md .agents/user_stories/*.md` should return no results.
|
||||
- `uv run ruff check README.md .agents/` — lint clean (markdown files are not linted by ruff, but the command should not error).
|
||||
|
||||
## Files changed
|
||||
|
||||
- `README.md` — Agent document tools section
|
||||
- `.agents/user_stories/agent-document-tools.md` — Acceptance criteria item 3
|
||||
Reference in New Issue
Block a user