chore(agent): phase roadmap from TODO.md, 7 phases (127-133)
Converts the TODO list into the phased-execution roadmap (Protocol B append): incremental folder summaries on sync (127), the mobile sync label with the pinned (n/m) count (128), per-chunk embed progress in the sync label (129), chat answer clusters with the bottom-right actions row (130), read(offset/limit) line ranges for huge documents (131), visible chat image-upload feedback (132), and the garbage-PDF text gate with the page-screenshot vision fallback (133 — includes the diagnosed root cause of the hnb guide's pre-phase-126 binary import). Each phase carries its dedicated in-isolation Playwright suite; TODO.md is cleared (items now live in .agents/phases/todo/). LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,51 @@
|
||||
# Phase 131 — `read` gains line ranges: grep finds the line, read(offset, limit) reads it
|
||||
|
||||
**Source:** `TODO.md` L7 — "The agent is struggling with huge documents. It can search for the text it wants and it finds the text, including line number, but then it gets stuck saying 'However, read takes a path, not a line number.' and can't figure out how to read that part of the document. Does the read tool support that? If not, it should. If it does, improve the description so the agent doesn't get stuck. I have a huge test pdf (the DND player's handbook) in test_sources called 'pdf_tests.tar' that you can use for testing. It's a massive pdf. The question the agent got stuck on was 'What are the components of the find traps spell?' which is toward the end of the players handbook so it will need to search and read specific lines. I've been setting BOR_READ_MAX_CHARS to 24000 to keep things relatively quick so the agent realizes quickly its answer isn't in the first few pages."
|
||||
(LOCKED A2, owner-confirmed 2026-09-27: the parameter names are `offset`/`limit` — 1-based line number + line count — mirroring the pi-harness `read` tool the model was trained on, the phase-70 "harness-aligned surface" anchor.)
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** `app/rag/agent.py` — `AGENT_TOOLS` (L258+: the `ls`/`read`/`grep` pi.dev-shaped schemas; the `read` schema at ~L304 takes ONLY `path`; the `grep` description promises `source/path:line: text` locators), `_execute_tool` (L1266+, the `read` branch at ~L1348: `arg` resolution, the `ALREADY_IN_CONTEXT` dedupe on `holder.read_docs`, the `BOR_READ_MAX_CHARS` truncation with `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` (~L506 — today it points at `grep` and then dead-ends), the `holder.read_truncations` tuples), `AgentHolder` (the `read_docs` / `read_truncations` bookkeeping); `app/rag/llm.py` — `ToolCallPiece` (the `arguments` dict the model sends); `tests/e2e/mock_llm.py` — the scripted tool-call flows (the "use your tools" / multi-read / grep patterns — the module header documents each trigger); `tests/unit/test_agent.py` — the read-branch unit suite (the `ALREADY_IN_CONTEXT` pins, the truncation pins); `test_sources/players_handbook.pdf` (the DND player's handbook from `pdf_tests.tar` — on disk, untracked); `settings.read_max_chars` (`BOR_READ_MAX_CHARS`).
|
||||
|
||||
## Objective
|
||||
The agent stops dead-ending on huge documents: `read` accepts an optional 1-based line `offset` + line-count `limit`, the truncation notice and tool descriptions teach the grep → read(offset/limit) workflow, and the dedupe rules allow the ONE case that matters — re-reading a range of a document whose first read was truncated — while still refusing pointless re-reads of text already in context. The owner's repro (the handbook, "find traps", `BOR_READ_MAX_CHARS=24000`) passes as an integration test.
|
||||
|
||||
## Dependencies
|
||||
- `130_chat_answer_clusters` (todo) — pipeline order only; code dependencies are the completed agent surface (phases 70/72/95/118).
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Schema + teaching (task 01):** `read` gains two OPTIONAL integer params — `offset` ("The first line to read, 1-based. Omit it to read from the beginning of the document.") and `limit` ("The maximum number of lines to read. Omit it to read as many lines as fit under the character cap."). `required` stays `["path"]`. The `read` description keeps its existing teaching and GAINS the workflow: for a very large document, use `grep` to find where your text is (it reports line numbers), then `read` that part with `offset` + `limit`; re-reading a document already read in full adds nothing. `READ_TRUNCATION_NOTICE` (L506) is REWRITTEN to end the dead-end: "…The rest is NOT shown. Use grep (pattern) to find what you need — it reports line numbers — then read that part with `offset` (1-based first line) and `limit` (number of lines)." The `grep` description gets the light touch: "read the winner with `read` — for a large document, use the reported line number as `offset`."
|
||||
- **Execution (task 02):** in `_execute_tool`'s read branch:
|
||||
- Validation: `offset`/`limit`, when present, must be ints ≥ 1 (NOT bools — the house `isinstance(x, int) and not isinstance(x, bool)` guard); a bad value → a teaching refusal echoing the bad value ("`offset` and `limit` are whole line numbers ≥ 1 — you passed `…`") — counts in nothing, consumes a round (the phase-72 refusal style).
|
||||
- The dedupe matrix (`known` = the `read_docs` (source, path) set; `truncated` = the set of (source, path) from `holder.read_truncations`):
|
||||
| call | doc in `known` | previously truncated | result |
|
||||
|---|---|---|---|
|
||||
| plain `read(path)` | no | — | full read (truncate at the cap as today) |
|
||||
| plain `read(path)` | yes | yes | NEW teaching line (no re-emit, no count): "You already have the first {shown} characters of {source}/{path} in your context. To read the rest, find where your text is with grep (it reports line numbers), then call read with `offset` (1-based first line) and `limit` (lines)." — `{shown}` from the matching `read_truncations` tuple |
|
||||
| plain `read(path)` | yes | no | `ALREADY_IN_CONTEXT` (byte-identical to today) |
|
||||
| `read(path, offset, limit)` | yes | yes | ALLOWED (the stuck-scenario fix — the range the model names is exactly the part it does NOT have) |
|
||||
| `read(path, offset, limit)` | yes | no | `ALREADY_IN_CONTEXT` (the full text is already in context — the range is a subset) |
|
||||
| `read(path, offset, limit)` | no | — | ALLOWED (a range read of an unread doc) |
|
||||
- `offset > total_lines` (the doc's `len(content.splitlines())`) → teaching refusal: "{source}/{path} is {total_lines} lines long — `offset` {offset} is past the end. Use a smaller offset (grep reports the exact line numbers)." Counts in nothing.
|
||||
- Result shape: line 1 `Document {source}/{path}:` + line 2 `date: YYYY-MM-DD` stay BYTE-IDENTICAL for plain reads (the E2E mock's `_READ_RESULT_PREFIX` contract); a RANGE read inserts a third line `lines: {offset}-{end} of {total_lines}` (the actual end = min(offset + limit − 1, total_lines); the model sees the true extent), then the image-doc marker (as today), then the slice (`"\n".join(lines[offset-1 : offset-1+limit])`).
|
||||
- The cap still applies to the SLICE: `len(slice) > settings.read_max_chars` → cut + `TRUNCATION_MARKER` + the (rewritten) notice; the `read_truncations` tuple for a range read is `(arg, cap, len(slice))` — the slice's requested size (documented: for range reads the tuple's `chars_total` is the slice length, not the document's).
|
||||
- A range read appends the doc to `read_docs` ONCE (the `known` set prevents a re-append) — so a later plain read of a range-read doc is `ALREADY_IN_CONTEXT` (the model has committed to this doc; the range result's `lines:` line told it what it holds).
|
||||
- **Handbook repro (task 03):** an integration test that imports `test_sources/players_handbook.pdf` as a local source (the mock LLM's deterministic embed/summary fakes — the in-process `tests/fakes.py` pattern; the handbook is ~1.5 MB of text ≈ 1200 chunks — in-process embedding is fast, keep the test under ~60 s) and drives the agent turn "What are the components of the find traps spell?" with `BOR_READ_MAX_CHARS=24000`: a scripted fake (the `test_agent.py` pattern) — round 1 → a `grep` tool call (pattern "find traps"), round 2 (the grep result carries the real line number) → a `read(path, offset=<that line>, limit=40)` call, round 3 → the answer echoing the read result's content (the phase-94 "read document reached the context" echo convention). Assert the echo contains the handbook's actual find-traps text (grep it from the extracted PDF text in the test — do not hardcode the handbook wording).
|
||||
- **Mock flow + E2E (task 04):** `tests/e2e/mock_llm.py` gains ONE scripted flow (house style — a user-message marker, deterministic reply, module-header docstring entry): the marker `read the lines` → request 1: a `grep` tool call (a canned pattern); request 2 (a `tool`-role grep result with a `source/path:line:` line present): a `read` tool call with `offset` = the parsed line number and `limit` = 30; request 3 (the read result present): the answer "Found it at line N." + the first 80 chars of the read result (the existing read-echo shape). `tests/e2e/test_read_line_ranges.py` (dedicated, in isolation, mock-only): a KB fixture with ONE large doc (> 24000 chars, the marker line `TRAP_COMPONENTS_LINE` near line ~1500), `BOR_READ_MAX_CHARS=24000`; ask a question containing the marker → assert the SSE `tool` frames (grep, then read with `offset`/`limit` args) and the final answer echoing the marker line; a second test: a plain read of the large doc truncates with the REWRITTEN notice (the notice names `offset`/`limit` — assert the `tool_result`/result surface carries it).
|
||||
|
||||
## Tasks
|
||||
1. `01_tool_surface.md` — the schema, the descriptions, the rewritten truncation notice.
|
||||
2. `02_range_read_execution.md` — the read branch: validation, the dedupe matrix, the slicing, the result shape.
|
||||
3. `03_handbook_repro.md` — the integration repro (the handbook + the owner's exact question) + the unit matrix.
|
||||
4. `04_mock_flow_e2e.md` — the mock-LLM `read the lines` flow + the dedicated E2E.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the full dedupe matrix (6 rows), the validation refusals, the slice math (boundaries, offset past EOF, 1-line docs), the result shapes (plain reads byte-identical — the existing pins must pass unchanged), the cap on slices.
|
||||
- Integration: the handbook repro (the owner's scenario, the owner's `BOR_READ_MAX_CHARS=24000`).
|
||||
- E2E: one dedicated Playwright file, run in isolation, mock-only.
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `read(path, offset, limit)` returns exactly the named lines with the `lines: a-b of N` header; plain reads are byte-identical to pre-phase
|
||||
- [ ] a truncated first read followed by a range read SUCCEEDS (the stuck scenario is gone); every other re-read is refused with the teaching
|
||||
- [ ] the handbook integration test answers the find-traps question from the real PDF text (no hardcoded wording)
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_read_line_ranges.py -v --no-cov` green in isolation
|
||||
- [ ] no behavior change in completed phases (the grep/ls surfaces, the plain-read contract, the `ALREADY_IN_CONTEXT` full-read refusal)
|
||||
@@ -0,0 +1,43 @@
|
||||
# Task 01 — The tool surface: `offset`/`limit` on `read`, the descriptions, the rewritten notice
|
||||
|
||||
**Phase:** `131_read_line_ranges` · **Source:** `TODO.md:7` — "The agent is struggling with huge documents. It can search for the text it wants and it finds the text, including line number, but then it gets stuck saying 'However, read takes a path, not a line number.' ... Does the read tool support that? If not, it should. If it does, improve the description so the agent doesn't get stuck."
|
||||
|
||||
## Objective
|
||||
The model's contract for `read` names the line-range capability (LOCKED A2: `offset`/`limit`, 1-based, pi-harness shape) and the truncation dead-end is rewritten into the working workflow: grep for the line numbers, then read that range.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — `AGENT_TOOLS`, the `read` entry (~L304): add to `properties`:
|
||||
```python
|
||||
"offset": {
|
||||
"type": "integer",
|
||||
"description": (
|
||||
"The first line to read, 1-based. Omit it to read from the "
|
||||
"beginning of the document. Use it to read a specific part "
|
||||
"of a large document — grep reports the line numbers of "
|
||||
"what you found."
|
||||
),
|
||||
},
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"description": (
|
||||
"The maximum number of lines to read, starting at offset. "
|
||||
"Omit it to read as many lines as fit under the character "
|
||||
"cap. Keep it small — you are reading a part, not the "
|
||||
"document."
|
||||
),
|
||||
},
|
||||
```
|
||||
`required` stays `["path"]`.
|
||||
2. `app/rag/agent.py` — the `read` `description`: keep the existing sentences (the `<documents>`-summaries teaching, the no-re-read rule, the truncation/`grep` paragraph — but REPLACE the truncation paragraph with the workflow): "Very large documents are truncated: you receive the first part plus a TRUNCATED notice naming how many more characters exist. To read the rest: use `grep` (pattern) to find where your text is — it reports line numbers — then call `read` again on the same document with `offset` (1-based first line) and `limit` (number of lines) to read exactly that part. Do not re-read a document you have already read in full — its full text is already in your prompt."
|
||||
3. `app/rag/agent.py` — `READ_TRUNCATION_NOTICE` (~L506): rewrite the tail (keep the `TRUNCATED — this document is {total} characters; only the first {shown} are in your context. The rest is NOT shown.` opening): "Use grep (pattern) to find what you need — it reports line numbers — then read that part with `offset` (1-based first line) and `limit` (number of lines)."
|
||||
4. `app/rag/agent.py` — the `grep` `description`: the "read the winner with `read`" sentence becomes "read the winner with `read` — for a large document, use the reported line number as `offset`".
|
||||
5. `tests/unit/test_agent.py` — update/extend the `AGENT_TOOLS` schema pins: the `read` schema has `offset`/`limit` (integer, optional — `required` is still `["path"]`), the descriptions contain the `offset`/`limit` teaching (the pinned substrings), and `READ_TRUNCATION_NOTICE` names both parameters.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the schema/notice pins above green; the `ls`/`grep` schemas' OTHER pins unchanged (byte-identical where not rewritten).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a model reading `AGENT_TOOLS` knows: grep → line number → `read(offset, limit)` (schema + descriptions + notice all agree)
|
||||
- [ ] the old dead-end sentence ("use grep to locate the rest" with no way to act on the line number) is gone from the notice
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,49 @@
|
||||
# Task 02 — Range-read execution: validation, the dedupe matrix, the slice, the result shape
|
||||
|
||||
**Phase:** `131_read_line_ranges` · **Source:** `TODO.md:7` — "The agent is struggling with huge documents. ... then it gets stuck saying 'However, read takes a path, not a line number.' and can't figure out how to read that part of the document."
|
||||
|
||||
## Objective
|
||||
`_execute_tool`'s read branch executes `offset`/`limit`: it validates the arguments with teaching refusals, applies the six-row dedupe matrix (the truncated-then-range case is the one that must succeed), slices the document's lines, and shapes the result so plain reads stay byte-identical while range reads carry their `lines: a-b of N` extent.
|
||||
|
||||
## Work
|
||||
1. `app/rag/agent.py` — `_execute_tool`, the `read` branch (~L1348): after `arg` is resolved and before the dedupe check, parse the range args:
|
||||
```python
|
||||
offset = call.arguments.get("offset")
|
||||
limit = call.arguments.get("limit")
|
||||
```
|
||||
with a helper `_read_range_args(offset, limit)` returning `(ok, error_line, offset, limit)`:
|
||||
- absent/None → ok, `(1, None)` semantics = plain read (no range);
|
||||
- present but not `isinstance(x, int) and not isinstance(x, bool)`, or `< 1` → the teaching refusal: "`offset` and `limit` are whole line numbers — pass integers ≥ 1 (you passed offset={offset!r}, limit={limit!r})." (echo the bad values; counts in nothing, consumes a round — the phase-72 refusal style, a new pinned constant `BAD_RANGE_ARGS`);
|
||||
- ok → the (offset, limit) pair (either may be None — `offset` without `limit` reads to the end, `limit` without `offset` is treated as `offset=1` — document both in the constant's docstring).
|
||||
2. `app/rag/agent.py` — the dedupe matrix (the phase-131 design table; `truncated_pairs = {(source, path) of holder.read_truncations}` — build it from the holder's tuples, matching on the RESOLVED pair, not the raw arg string):
|
||||
- plain read + in `known` + truncated → the new pinned teaching line `REREAD_TRUNCATED` ("You already have the first {shown} characters of {source}/{path} in your context. To read the rest, find where your text is with grep (it reports line numbers), then call read with `offset` (1-based first line) and `limit` (lines).") — `{shown}` from the matching tuple; no re-emit, `tool_calls` NOT incremented (a refusal);
|
||||
- plain read + in `known` + not truncated → `ALREADY_IN_CONTEXT` (the existing byte-identical path — do not touch that constant);
|
||||
- range read + in `known` + truncated → PROCEED (the fix);
|
||||
- range read + in `known` + not truncated → `ALREADY_IN_CONTEXT`;
|
||||
- range read + not in `known` → PROCEED.
|
||||
(The current code's dedupe check runs on the RAW arg before resolution for speed — keep that fast path for the plain-read `ALREADY_IN_CONTEXT` case; the truncated-set check needs the resolved pair, so it runs after `_resolve_path`, exactly where the `doc` is known. Restructure minimally: the raw-arg fast path handles "plain read + known"; everything else resolves first, then consults the matrix.)
|
||||
3. `app/rag/agent.py` — the slice + result (after `doc` is resolved and the matrix passed):
|
||||
- `lines = doc.content.splitlines()`; `total_lines = len(lines)`;
|
||||
- `offset > total_lines` → the teaching refusal `OFFSET_PAST_END` ("{source}/{path} is {total_lines} lines long — `offset` {offset} is past the end. Use a smaller offset (grep reports the exact line numbers).") — no count;
|
||||
- `end = min(offset + (limit or 1) - 1, total_lines)`; `slice_lines = lines[offset - 1 : end]`; `slice_text = "\n".join(slice_lines)`;
|
||||
- `holder.read_docs.append(doc)` (the `known` check already guarantees no duplicate); `holder.tool_calls += 1` (a range read is a successful call — the plain-read convention);
|
||||
- result: plain read → EXACTLY today's bytes (header, date, marker, content/truncated-content — the existing code path untouched); range read →
|
||||
```
|
||||
Document {source}/{path}:
|
||||
date: YYYY-MM-DD
|
||||
lines: {offset}-{end} of {total_lines}
|
||||
{marker}{slice_text}
|
||||
```
|
||||
(the `lines:` line replaces the position the marker/`TRUNCATION_MARKER` occupy in the plain read — for a range read the marker line comes after `lines:` exactly as after the date line in the plain read);
|
||||
- the cap on the slice: `len(slice_text) > settings.read_max_chars` → `slice_text[:cap]` + `\n{TRUNCATION_MARKER}\n{READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text))}` and `holder.read_truncations.append((arg, cap, len(slice_text)))` — for range reads the tuple's `chars_total` is the SLICE's requested size (document in the `AgentHolder.read_truncations` field comment: "for range reads the slice length, not the document's").
|
||||
4. `tests/unit/test_agent.py` — the full matrix (six rows, each pinned): the six dedupe outcomes; `BAD_RANGE_ARGS` (a float, a string, a bool, a 0, a negative); `OFFSET_PAST_END` (offset = total+1 on a 10-line doc); the slice math (offset=1/limit=None → the whole doc; offset=total → the last line; limit overshoot → `end` clamps to `total_lines`; a 1-line doc; offset past EOF); the result shapes (the plain-read pins pass UNCHANGED — byte-identical; a range read's `lines: a-b of N` line + slice; the cap on a huge slice + the tuple's slice-length `chars_total`).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the matrix + the pins above — this task's suite.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the six-row matrix is pinned one test per row (the truncated-then-range row PASSES — the stuck scenario's unit proof)
|
||||
- [ ] plain reads are byte-identical to pre-phase (the existing pins green, unmodified)
|
||||
- [ ] a range read's result names its true extent (`lines: a-b of N`) and respects the cap
|
||||
- [ ] `uv run pytest tests/unit -q` green; no behavior change in completed work
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 03 — The handbook repro: "find traps" on the real D&D player's handbook
|
||||
|
||||
**Phase:** `131_read_line_ranges` · **Source:** `TODO.md:7` — "I have a huge test pdf (the DND player's handbook) in test_sources called 'pdf_tests.tar' that you can use for testing. It's a massive pdf. The question the agent got stuck on was 'What are the components of the find traps spell?' which is toward the end of the players handbook so it will need to search and read specific lines. I've been setting BOR_READ_MAX_CHARS to 24000 to keep things relatively quick so the agent realizes quickly its answer isn't in the first few pages."
|
||||
|
||||
## Objective
|
||||
The owner's exact incident is an integration test: the real `players_handbook.pdf` is indexed, the agent is asked the find-traps question with the owner's `BOR_READ_MAX_CHARS=24000`, and the scripted agent loop performs the grep → read(offset/limit) dance against the REAL document and lands an answer grounded in the handbook's actual find-traps text.
|
||||
|
||||
## Work
|
||||
1. `tests/integration/test_read_line_ranges_handbook.py` (new) — the house integration pattern (inspect `tests/integration/` for the agent-loop test that runs a scripted fake LLM against a real-ish pipeline — `tests/unit/test_agent.py`'s fake-LLM/scripted-rounds pattern is the model; lift it into an integration test that ALSO runs the real PDF extraction + real chunking/embedding fakes from `tests/fakes.py`):
|
||||
- Fixture: a temp source dir containing `test_sources/players_handbook.pdf` (copy — never mutate the untracked fixture), registered/imported with the REAL importer (`import_sources`, the mock embed/summary fakes — the in-process fakes make the ~1200-chunk embed fast; keep the whole test under ~60 s; if the import is slower, the test may pre-import once at module scope and share it across its tests).
|
||||
- The scripted fake (deterministic, the `test_agent.py` round pattern):
|
||||
- round 1 (no tool results yet) → a `grep` tool call, `pattern="find traps"` (lowercase — grep is case-insensitive);
|
||||
- round 2 (the grep result carries `…:LINE: …find traps…` — parse the REAL line number from the grep result the loop produced, do not hardcode it) → a `read` tool call: `path` = the handbook's `source/path`, `offset` = the parsed line, `limit` = 40;
|
||||
- round 3 (the read result present) → the answer: the read result's content, verbatim (the phase-94 "the read document reached the context" echo convention — the loop's reply IS the slice).
|
||||
- Settings: `BOR_READ_MAX_CHARS=24000` (the owner's env — the handbook is far longer, so round 1's plain-read alternative would truncate; the script goes straight to grep, but the setting must be in force for the notice/teaching path to be realistic).
|
||||
2. Assertions (the observable contract):
|
||||
- the final answer contains the handbook's REAL find-traps text — derive the expected substring at test time: extract the PDF's text (the phase-126 `extract_pdf_text`), grep it for the find-traps passage in the test itself (the same way the model does), and assert the ANSWER contains that passage's first 60 chars (NO hardcoded handbook wording — the test stays true if the fixture is re-extracted);
|
||||
- the agent's `read` call carried `offset`/`limit` (assert on the fake's captured round-2 tool call: `offset == the grep-reported line`, `limit == 40`);
|
||||
- the loop made exactly 3 rounds (grep → range read → answer) with `tool_calls == 2` (the owner's stuck loop made 2 and dead-ended — the fixed loop closes in 3).
|
||||
3. `tests/integration/test_read_line_ranges_handbook.py` — a SECOND test, the plain-read-first variant: the fake round 1 does a PLAIN `read(path)` (the handbook truncates at 24000 with the rewritten notice), round 2 does `read(path, offset=<grep line>, limit=40)` AFTER a grep — asserting the truncated-then-range matrix row works on the REAL document (the `REREAD_TRUNCATED` teaching is never needed because the model went straight to the range — the matrix row itself is unit-pinned in task 02; here the end-to-end path is the point).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: this file IS the task's suite (plus task 02's unit matrix).
|
||||
- Coverage: **>90%** on new/modified code (the test file is covered by the suite run; `app/` changes are task 02's).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/integration/test_read_line_ranges_handbook.py -v` green in ~60 s (the owner's scenario passes on the real PDF)
|
||||
- [ ] the expected text is derived from the fixture at test time (grep the extracted text — no hardcoded handbook wording)
|
||||
- [ ] the read call's `offset` is the grep-reported line (parsed, not literal)
|
||||
- [ ] no behavior change in completed phases
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 04 — The mock flow `read the lines` + the dedicated E2E
|
||||
|
||||
**Phase:** `131_read_line_ranges` · **Source:** `TODO.md:7` — "The agent is struggling with huge documents. ... Does the read tool support that? If not, it should. If it does, improve the description so the agent doesn't get stuck."
|
||||
|
||||
## Objective
|
||||
The E2E mock deterministically performs the grep → read(offset/limit) dance, and a dedicated, in-isolation Playwright suite pins the full SSE surface: the `tool` frames carry the range args, the answer is grounded in the large doc's tail, and the truncation notice now teaches the workflow instead of dead-ending.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/mock_llm.py` — one new scripted flow in the house style (a user-message marker + a module-header docstring entry, exactly like the existing "use your tools" / multi-read / grep flows — read the module header first and match its format):
|
||||
- Marker: the user message containing `read the lines` (a dedicated trigger — no overlap with the existing grep-flow trigger; if the existing grep flow's trigger could also match, make the new marker more specific, e.g. `read the lines of` — the header documents the exact string).
|
||||
- Flow (classified by the count of `tool`-role results in the messages, the house pattern):
|
||||
- 0 results → a `grep` tool call, `pattern="LINE_MARKER"` (a canned constant the fixture doc contains);
|
||||
- a `tool`-role grep result present (a line matching `^[\w/]+:(\d+):` — parse the FIRST match's line number) → a `read` tool call: `path` = the grep result's `source/path`, `offset` = the parsed line number, `limit` = 30 (structured `tool_calls`, no reasoning — the terminal-shape convention);
|
||||
- the read result present (a `tool`-role result whose content starts with `Document `) → the forced answer, byte-stable: `Found it at line N.` (N = the offset the model passed — parse it from the `read` call's arguments in the messages) + the first 80 chars of the read result's content (the phase-94 read-echo convention — proves the RANGE reached the model's context).
|
||||
2. `tests/e2e/test_read_line_ranges.py` (new) — dedicated, in isolation, mock-only, house pattern (module app fixture, temp local source, autouse cleanup):
|
||||
- KB fixture: ONE large markdown doc `linedoc/big.md` — ~100 KB, generated deterministically in the test (numbered filler lines `Line <i>: …`, so line numbers are exact), with the marker line `LINE_MARKER the trap components are documented here` at a known line ~1500 (far beyond the 24000-char cap — the doc's first 24000 chars hold NO marker);
|
||||
- app env: `BOR_READ_MAX_CHARS=24000` (the owner's setting — the plain read MUST truncate).
|
||||
- Test → observable mapping (Playwright Mapping Rule):
|
||||
- `test_range_read_flow_over_sse` — send a question containing `read the lines` → the SSE `tool` frames, in order: the `grep` call (args `pattern="LINE_MARKER"`), then the `read` call (args: `path` = the doc's `source/path`, `offset` an int == the marker's true line, `limit` == 30 — assert on the frame's args, not a substring); the `done` frame's answer contains `Found it at line <N>` (N == the marker's line) and the marker text's first 80 chars (the read-echo — the range reached the context); NO truncation-notice dead-end (the answer is the grounded echo, not a "I can't read that part" shape — assert the marker text is present).
|
||||
- `test_plain_read_truncation_notice_teaches_the_workflow` — a SECOND trigger (reuse the mock's existing plain-read flow on this large doc — the `read` of `linedoc/big.md` truncates at 24000): the surfaced truncation (the `tool_result` frame / the result the model sees — assert on the SSE `tool_result` surface per A15) contains the REWRITTEN notice text naming `offset` and `limit` (the pinned substrings `"offset"` + `"limit"` in the notice — the old dead-end wording is gone).
|
||||
3. Header docstring: the run-in-isolation command (`uv run pytest tests/e2e/test_read_line_ranges.py -v --no-cov`), the mock-only gate, the fixture's determinism (the generated line numbers), the `BOR_READ_MAX_CHARS=24000` rationale (the owner's env), the test→observable table.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: this file IS the task's suite — run in isolation, green.
|
||||
- Coverage: the phase's >90% `app/` bar is carried by tasks 01–02; the handbook integration repro by task 03; this task adds the in-isolation Playwright gate.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_read_line_ranges.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`)
|
||||
- [ ] the `read` tool frame carries REAL `offset`/`limit` ints (parsed from the grep result — the mock does the owner's exact dance)
|
||||
- [ ] the truncation notice surfaces the workflow teaching (the dead-end is gone)
|
||||
- [ ] no behavior change in completed phases (the existing mock flows' triggers still fire unchanged)
|
||||
Reference in New Issue
Block a user