phase: 125_image_anchor_context
All completion criteria verified. The repository is left functional with all changes in the working tree (no commits made, per the harness protocol). ## Final report — phase 125 `image_anchor_context` (verification pass) All 6 tasks were already complete; this pass verified every criterion and ran the full gates: - **Migration 0023** `documents.image_context` (TEXT nullable, reversible) — applied (`alembic current` = 0023 head); round-trip test (4 passed) exercises real upgrade → downgrade 0022 → upgrade. - **Criterion 1** (referenced image enriched + context line in retrieval): integration `test_markdown_referenced_image_gets_context_and_enriched_chunks` + E2E `test_sync_enriches_the_image_doc_with_markdown_context`, `test_chat_answer_quotes_the_context_line` (wire prompt pinned: `<document>` block leads with the one-line `Referenced in …: "alt"` line) — verified present and passing. - **Criterion 2** (standalone image byte-identical): E2E `test_standalone_*` (NULL `image_context`, block body exactly the bare description, no `Referenced in` anywhere in the prompt) + unit `test_describe_image_without_context_is_byte_identical` — passing. - **Criterion 3** (markdown-first: record stored on markdown doc, image picks up context later): integration `test_markdown_indexed_before_image_stores_record_and_image_picks_it_up` + post-index enrichment pass test (markdown file gone from disk) — passing. - **Gates:** `uv run pytest` → 2938 passed, exit 0 · `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL **99%** (>90%) · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors, 0 warnings. - **Phase E2E in isolation:** `uv run pytest tests/e2e/test_image_anchor_context.py -v --no-cov` → 4 passed. - **Mock-LLM change regression check:** neighboring image E2E suites green in isolation — `test_smoke` (3), `test_image_documents` (16), `test_chat_image_questions` (17), `test_answer_images` (17). - No defects found; no code changes needed. No live-infrastructure changes (local dev DB only). No commit made (harness commits). - Next pending phase: check `.agents/phases/todo/` after the harness moves this phase.
This commit is contained in:
@@ -0,0 +1,54 @@
|
||||
# Phase 125 — Image-anchor context: markdown image references enrich image documents with surrounding text
|
||||
|
||||
**Source:** `TODO.md` — "improve image handling: use context of surrounding text to describe images, use alt text from markdown image syntax `` to give context to the summary."
|
||||
**Story:** n/a (improvement on phase-122/124 image capability).
|
||||
**Context:** `app/config.py` — `Settings` (`images` L379, `image_extensions` L390, `image_dir` L413). `app/rag/importer.py` — `iter_importable_files` (L235, extension walk), `import_sources` (L279), `_index_file` (L435 — text branch for markdown, binary branch delegates to `_index_image_file`), `_index_image_file` (L801 — bytes digest, persistent copy, `describe_image` seam, chunk + embed pipeline), `_store_summary` (L582), `_describe_or_skip` (L762 — calls `describe_image`). `app/rag/summarizer.py` — `describe_image` (L208 — multimodal chat call, DESCRIBE_PROMPT constant), `DESCRIBE_INSTRUCTION` (L89 — the locked prompt text), `generate_summary` (L182), `build_summary_prompt` (L162). `app/rag/chunker.py` — markdown section splitter on `##` headings, paragraph packing. `app/models.py` — `Document` (L102: `content`, `summary`, `is_image`, `image_path`), `Chunk` (L171). `app/rag/retriever.py` — `_selection_order` (L666, shared selection walk), `select_suggested` (L864, seeds prompt summaries), `source_ref_with_image` (L945). `app/rag/prompts.py` — `<document>` block body (L438, uses `doc.summary`), `build_high_prompt` (L461 — the LOCKED `<tools>` section, phase-124 `show_image` clause). `app/api/chat.py` — done-frame build (L1119-1133, `cited_docs` merge from task 05 of phase 124). `alembic/versions/0022_documents_image.py` — the phase-122 migration (the pattern for additive, reversible migrations).
|
||||
|
||||
## Objective
|
||||
When a markdown file references an image (e.g. ``), the image document's description is enriched with the alt text and surrounding paragraphs — so the image becomes retrievable not just by what it shows, but by why it exists in the documentation. This closes the gap between standalone image descriptions (phase 122) and the rich contextual text that lives in the parent markdown document.
|
||||
|
||||
## Dependencies
|
||||
- `122_image_documents` (complete) — the image-doc pipeline: `Document.is_image`/`image_path`, `describe_image`, the binary index path, `BOR_IMAGES` toggle, the vision description as content+summary.
|
||||
- `124_image_chips_and_answers` (complete) — auto-cite for image docs, `BOR_ANSWER_IMAGES` toggle, the `show_image` tool.
|
||||
- Code dependencies (all complete): phase 30 summary pipeline, phase 118 summary seeding, phase 119 read-docs citation tier.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **The `image_context` field (task 01):** `Document.image_context: str | None = None` (TEXT, nullable). Stores the surrounding text extracted from the markdown file that references the image — alt text + N paragraphs before and after the reference. NULL for image docs that were not found to have a markdown reference (standalone images, images referenced in non-markdown formats, pre-phase rows). Migration: one additive, reversible migration (A13 precedent: `0023_documents_image_context.py`).
|
||||
- **Markdown image extraction (task 02):** a new module-level function `extract_image_refs(content: str) -> list[ImageRef]` in `app/rag/importer.py` (or `app/rag/md_images.py` if preferred) that parses markdown content and extracts: (a) the `` pattern via regex; (b) for each reference, the N paragraphs before and after the line containing the reference (configurable, default 2); (c) the line number of the reference for context window calculation. The regex: `r'!\[([^\]]*)\]\(([^)]+)\)'` — captures alt text and the image path. The context window: split content into paragraphs (blank-line separated, respecting code fences), find the paragraph containing the reference, take N before and N after. Result: `ImageRef(path, alt_text, context_text, line_number)`.
|
||||
- **Enrichment flow (task 03):** after `_index_file` processes a markdown file, check if it contains image references. For each reference whose image path resolves to an existing image document in the same source, load the image doc and enrich its content: `new_content = f"{alt_text}\n\n{context}\n\n{original_description}"` — the alt text first (short, purposeful), then the surrounding context (why the image exists), then the vision description (what the image shows). Then re-chunk and re-embed the enriched content. If the image doc doesn't exist yet (not indexed), store the reference on the markdown doc's `image_context` field so the enrichment can happen when the image is later indexed.
|
||||
- **Enrichment at image-index time (task 04):** in `_index_image_file`, BEFORE calling `describe_image`, check if any markdown file in the same source references this image. If so, load the markdown content, extract the reference + context, and pass it to `describe_image` as additional context in the prompt (the vision model can acknowledge the context: "This diagram, referenced in `<markdown_path>` as `<alt_text>`, shows..."). This is the primary enrichment path — the image gets context during its own indexing. The `image_context` field on the image doc stores the extracted context text for retrieval display.
|
||||
- **Retrieval display (task 05):** when an image doc is in the suggested tier, the `<document>` block (which currently shows `doc.summary`) is extended to also show the `image_context` (if present) — a short "Referenced in `<source>/<markdown_path>`: `<alt_text>`" line before the description. This is a small addition to `build_document_block` in `app/rag/prompts.py` — only for `is_image` docs with non-NULL `image_context`.
|
||||
- **NOT touched:** the `BOR_IMAGES` toggle (the feature is gated by it, unchanged), the `BOR_ANSWER_IMAGES` toggle (unchanged), the `show_image` tool (unchanged), the image serve route (unchanged), the document viewer (unchanged), and the prune logic (unchanged). Existing image docs without context are unaffected — `image_context` is NULL, the prompt block shows only the description (byte-identical to pre-phase).
|
||||
- **Locked assumptions:** **A1** — `image_context` stores the surrounding text from the referencing markdown file. **A2** — enrichment happens at image-index time (the image doc picks up context from its markdown parent). **A3** — the vision model receives context in the prompt but the description remains the primary content. **A4** — retrieval display adds a short context line for enriched image docs; pre-phase image docs are byte-identical.
|
||||
|
||||
## Tasks
|
||||
1. `01_migration_image_context.md` — `Document.image_context` column + migration.
|
||||
2. `02_extract_image_refs.md` — `extract_image_refs()` function for markdown image reference extraction.
|
||||
3. `03_enrich_image_description.md` — pass context to `describe_image`; enrich image doc content + re-embed.
|
||||
4. `04_store_image_context.md` — store context on image docs; handle the case where markdown is indexed before/after the image.
|
||||
5. `05_retrieval_display.md` — show context line for enriched image docs in the `<document>` block.
|
||||
6. `06_image_anchor_tests.md` — unit + integration + E2E tests.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_image_anchor_context.py` (new, task 06) — `extract_image_refs` (basic reference, multiple references, references in code fences, references with no alt text, references in headings, edge cases); the enrichment pipeline (context extraction, content enrichment, re-chunking, re-embedding); the `describe_image` prompt extension (context included in the text part when provided); the retrieval display (context line present for enriched docs, absent for non-enriched).
|
||||
- Integration: `tests/integration/test_importer.py` (extend) — a markdown file referencing an image in the same source → the image doc has `image_context` set; the image description includes the context; the chunks are re-embedded with the enriched content. A markdown file referencing a non-existent image → the markdown doc's `image_context` is set (the image will be enriched when indexed). A standalone image (no markdown reference) → `image_context` is NULL (byte-identical to pre-phase).
|
||||
- E2E: `tests/e2e/test_image_anchor_context.py` (new, task 06) — isolated run: upload a fixture markdown file that references an image in the same archive → sync → the image doc has enriched context → ask a question that requires the context to answer correctly (the mock LLM should be able to use the context to give a better answer) → the `<document>` block shows the context line.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A markdown file referencing an image in the same source → the image document's description is enriched with alt text + surrounding context; the retrieval block shows the context line.
|
||||
- [ ] A standalone image (no markdown reference) → `image_context` is NULL; behavior byte-identical to pre-phase.
|
||||
- [ ] A markdown file referencing a non-existent image → the context is stored on the markdown doc; when the image is later indexed, it picks up the context.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
|
||||
|
||||
## Locked decisions
|
||||
- **A1 — `image_context` stores the surrounding text (alt text + N paragraphs before/after) from the referencing markdown file (owner-confirmed 2026-09-26, roadmap confirmation).**
|
||||
- **A2 — enrichment happens at image-index time: the image doc picks up context from its markdown parent (owner-confirmed 2026-09-26, roadmap confirmation).**
|
||||
- **A3 — the vision model receives context in the prompt but the description remains the primary content (owner-confirmed 2026-09-26, roadmap confirmation).**
|
||||
- **A4 — retrieval display adds a short context line for enriched image docs; pre-phase image docs are byte-identical (owner-confirmed 2026-09-26, roadmap confirmation).**
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ alembic/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): enrich image descriptions with markdown context (alt text + surrounding paragraphs)"
|
||||
```
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 01 — Migration: `Document.image_context` column
|
||||
|
||||
**Phase:** `125_image_anchor_context` · **Source:** `00_phase.md` — the `image_context` field
|
||||
|
||||
## Objective
|
||||
Add `Document.image_context: str | None` (TEXT, nullable) to the database schema — the column that stores the surrounding text extracted from the markdown file referencing the image. One additive, reversible migration (A13 precedent).
|
||||
|
||||
## Work
|
||||
1. `app/models.py` — add to `Document` (after `image_path`, L168):
|
||||
```python
|
||||
#: Surrounding text from the markdown file that references this image
|
||||
#: (phase 125, LOCKED A1): alt text + N paragraphs before/after the
|
||||
#: reference. NULL for standalone images and pre-phase rows. The
|
||||
#: embedding model embeds this alongside the vision description when
|
||||
#: the image is enriched (task 03).
|
||||
image_context: Mapped[str | None] = mapped_column(Text, default=None)
|
||||
```
|
||||
2. `alembic/versions/0023_documents_image_context.py` — new migration:
|
||||
- `revision = "0023"`, `down_revision = "0022"`
|
||||
- `upgrade()`: `op.add_column("documents", sa.Column("image_context", sa.Text(), nullable=True))`
|
||||
- `downgrade()`: `op.drop_column("documents", "image_context")`
|
||||
- Docstring following the `0022_documents_image.py` pattern: explain the column, its purpose, and that it's fully reversible.
|
||||
3. ASSUMPTION: none. The column is nullable (NULL for all pre-phase rows) and additive (no existing code breaks).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_models.py` (extend) — `Document(image_context="some context")` parses; `Document(image_context=None)` parses; the column is nullable (no NOT NULL constraint).
|
||||
- Integration: `tests/integration/test_migrations.py` (extend) — run `alembic upgrade head` (the new migration applies cleanly); run `alembic downgrade 0022` (the downgrade restores the 0022 schema); run `alembic upgrade head` again (idempotent). Assert that pre-phase image docs (NULL `image_context`) round-trip correctly.
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `alembic upgrade head` applies the new migration cleanly.
|
||||
- [ ] `alembic downgrade 0022` + `upgrade head` round-trips the schema.
|
||||
- [ ] `Document` model has the `image_context` field; NULL is valid.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Task 02 — Extract image references from markdown content
|
||||
|
||||
**Phase:** `125_image_anchor_context` · **Source:** `00_phase.md` — the `extract_image_refs()` function
|
||||
|
||||
## Objective
|
||||
A pure function that parses markdown content and extracts all `` references, each with its alt text, the surrounding paragraph context (N before and N after), and the line number. Returns a list of `ImageRef` dataclasses.
|
||||
|
||||
## Work
|
||||
1. `app/rag/md_images.py` — new module (or add to `app/rag/importer.py` if preferred — the executor's call, but a separate module keeps the importer lean):
|
||||
- `@dataclass class ImageRef: path: str; alt_text: str; context: str; line_number: int` — the extracted reference with its context window.
|
||||
- `CONTEXT_PARAGRAPHS: int = 2` — configurable number of paragraphs before/after (the default gives ~2-4 sentences of surrounding text).
|
||||
- `def extract_image_refs(content: str, context_paragraphs: int = CONTEXT_PARAGRAPHS) -> list[ImageRef]`:
|
||||
- Split content into lines.
|
||||
- Track code fence state (a fenced code block is NOT context — image references inside code fences are just text, not actual references).
|
||||
- For each line, check for the image reference regex: `r'!\[([^\]]*)\]\(([^)]+)\)'`.
|
||||
- When a reference is found:
|
||||
- Record the line number (1-indexed).
|
||||
- Find the paragraph containing this line (paragraphs are separated by blank lines, but code fences are atomic).
|
||||
- Collect N paragraphs before and N after the reference paragraph.
|
||||
- The `context` is the joined paragraphs (excluding the reference line itself, which is already captured as `alt_text` + `path`).
|
||||
- The `alt_text` is the captured group (may be empty).
|
||||
- The `path` is the captured group (may be relative or absolute).
|
||||
- Return the list of `ImageRef` objects.
|
||||
- Edge cases to handle:
|
||||
- References inside code fences → skip (the reference is literal text, not an actual image).
|
||||
- Multiple references on the same line → extract each separately.
|
||||
- References with no alt text (``) → `alt_text = ""`.
|
||||
- References with no surrounding text (start/end of file) → context is whatever is available.
|
||||
- Image paths with special characters (spaces, parentheses) → the regex should handle common cases; edge cases with nested parentheses in paths are acceptable to miss (they're rare in practice).
|
||||
2. ASSUMPTION: none. This is a pure function — no I/O, no LLM calls, no database access.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_md_images.py` (new, task 06) — test `extract_image_refs` against various inputs:
|
||||
- Single reference: `` → one `ImageRef` with alt="alt", context from surrounding paragraphs.
|
||||
- Multiple references: two `![...]` on different lines → two `ImageRef`s, each with its own context.
|
||||
- Reference in code fence: ````  ```` → skipped (not extracted).
|
||||
- Reference with no alt text: `` → `alt_text = ""`.
|
||||
- Reference at start of file → context is only paragraphs after.
|
||||
- Reference at end of file → context is only paragraphs before.
|
||||
- Reference in a heading: `## ` → extracted, but context is the surrounding text (the heading line itself is not included in context).
|
||||
- Empty content → empty list.
|
||||
- Content with no references → empty list.
|
||||
- Verify that `line_number` is 1-indexed and correct.
|
||||
- Coverage: **>90%** on the new module.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `extract_image_refs("")` returns one `ImageRef` with correct alt, path, context, and line number.
|
||||
- [ ] References inside code fences are skipped.
|
||||
- [ ] Multiple references are extracted separately with correct context windows.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Task 03 — Enrich image descriptions with context
|
||||
|
||||
**Phase:** `125_image_anchor_context` · **Source:** `00_phase.md` — pass context to `describe_image`; enrich image doc content + re-embed
|
||||
|
||||
## Objective
|
||||
When indexing an image file, check if any markdown file in the same source references it. If so, extract the alt text and surrounding context, and pass it to `describe_image` so the vision model can produce a context-aware description. Store the extracted context on the image doc's `image_context` field.
|
||||
|
||||
## Work
|
||||
1. `app/rag/summarizer.py` — extend `describe_image`:
|
||||
- Add an optional `context: str | None = None` parameter.
|
||||
- When `context` is provided, prepend it to the DESCRIBE_PROMPT text part:
|
||||
```python
|
||||
context_prefix = f"\n\nThis image is referenced in documentation with the following context:\n{context}" if context else ""
|
||||
messages = [{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": DESCRIBE_PROMPT + context_prefix},
|
||||
{"type": "image_url", "image_url": {"url": data_url}},
|
||||
],
|
||||
}]
|
||||
```
|
||||
- The context prefix teaches the vision model what the image is for, enabling a more useful description. The existing DESCRIBE_PROMPT (the locked instruction) is unchanged — the context is appended after it.
|
||||
- The context is NOT stored on the doc (that's task 04's job); this task only passes it to the LLM.
|
||||
2. `app/rag/importer.py` — in `_index_image_file`, BEFORE calling `describe_image`:
|
||||
- Check if any markdown file in the same source references this image. The simplest approach: scan all `.md`/`.markdown` files in the same source directory for references to this image's filename. Use `extract_image_refs` from task 02.
|
||||
- For each markdown file that references this image, collect the `ImageRef` objects where the image path matches (handle both relative paths like `image.png` and paths like `folder/image.png`).
|
||||
- If references are found, combine them into a single context string:
|
||||
```
|
||||
Referenced in <source>/<markdown_path>:
|
||||
- Alt text: "<alt_text>"
|
||||
Context: <context_paragraphs>
|
||||
```
|
||||
- Pass this context to `describe_image` as the `context` parameter.
|
||||
- Store the combined context string on the image doc's `image_context` field (this is the primary storage — the retrieval display in task 05 reads from here).
|
||||
3. `app/rag/importer.py` — in `_index_image_file`, after `describe_image` returns:
|
||||
- If `image_context` is set (from step 2), the image doc's `content` should be enriched: `content = f"{image_context}\n\n{description}"` — the context first (for retrieval), then the description. Then the normal chunk pipeline embeds this enriched content.
|
||||
- If `image_context` is NULL (no markdown reference found), the content is just the description (byte-identical to pre-phase).
|
||||
- The `summary` field: for image docs, `summary = content` (the phase-122 convention). So the summary is also enriched.
|
||||
4. ASSUMPTION: none. The context extraction uses the `extract_image_refs` function from task 02. The `describe_image` extension is backward-compatible (the `context` parameter defaults to `None`).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_image_anchor_context.py` (task 06) — `describe_image` with `context=None` → same behavior as pre-phase (the DESCRIBE_PROMPT is unchanged); `describe_image` with `context="some context"` → the prompt includes the context prefix; the enriched content is `context + "\n\n" + description`.
|
||||
- Integration: `tests/integration/test_importer.py` (extend) — seed a source with both a markdown file referencing an image and the image file itself → sync with `images=True` → the image doc has `image_context` set, its `content` includes the context, its chunks are embedded from the enriched content. A standalone image (no markdown reference) → `image_context` is NULL, `content` is just the description.
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] An image referenced in a markdown file → the description is context-aware (the LLM receives the context), and the image doc's `content` includes the context.
|
||||
- [ ] A standalone image → `image_context` is NULL, `content` is just the description (byte-identical to pre-phase).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,48 @@
|
||||
# Task 04 — Store context on markdown docs for future image enrichment
|
||||
|
||||
**Phase:** `125_image_anchor_context` · **Source:** `00_phase.md` — handle the case where markdown is indexed before/after the image
|
||||
|
||||
## Objective
|
||||
When a markdown file is indexed and references an image that doesn't yet exist as a document (e.g., the image hasn't been synced yet, or it's in a different source), store the image references on the markdown doc's `image_context` field. This enables a post-indexing enrichment pass that can pick up the context when the image is later indexed.
|
||||
|
||||
## Work
|
||||
1. `app/rag/importer.py` — in the markdown/text branch of `_index_file` (after the Document upsert, before the chunk pipeline):
|
||||
- After extracting the title (L505-509), check if the file is markdown (`full_path.suffix.lower() in (".md", ".markdown")`).
|
||||
- Call `extract_image_refs(content)` to get all image references.
|
||||
- For each reference, check if an image document already exists in the same source with a matching path. Use a database query: `select(Document).where(Document.source == source, Document.path == image_ref.path, Document.is_image == True)`.
|
||||
- For references where the image document DOES NOT exist, store the reference info on the markdown doc's `image_context` field:
|
||||
```python
|
||||
# Build a structured context string for references to non-existent images
|
||||
missing_refs = [r for r in image_refs if not image_exists_in_source(session, source, r.path)]
|
||||
if missing_refs:
|
||||
doc.image_context = _format_missing_image_refs(missing_refs, source)
|
||||
```
|
||||
- `_format_missing_image_refs(refs, source) -> str`: formats the missing references as a structured string:
|
||||
```
|
||||
IMAGE_CONTEXT:
|
||||
- [image.png] (line 42): "Alt text here" — 2 paragraphs of context
|
||||
- [diagram.jpg] (line 100): "" — 1 paragraph of context
|
||||
```
|
||||
This format is parseable by the enrichment pass (task 04b).
|
||||
- For references where the image document DOES exist, the enrichment happens in task 03 (at image-index time). The markdown doc's `image_context` remains NULL.
|
||||
2. `app/rag/importer.py` — a new helper `_format_missing_image_refs(refs, source) -> str` (module-level, pure function):
|
||||
- Takes a list of `ImageRef` objects and the source name.
|
||||
- Returns a formatted string as described above.
|
||||
- Used by both `_index_file` (markdown branch) and the enrichment pass.
|
||||
3. `app/rag/importer.py` — after `_index_file` returns for all files in a source (in `import_sources`), run a **post-indexing enrichment pass**:
|
||||
- Query all image documents in the source that have `image_context IS NULL` (not yet enriched).
|
||||
- For each, check if any markdown document in the same source has `image_context` set (contains references to this image).
|
||||
- If a match is found, extract the reference info from the markdown doc's `image_context`, enrich the image doc (same logic as task 03: re-call `describe_image` with context, update `content`, re-embed chunks).
|
||||
- This pass handles the case where the markdown was indexed first (storing references) and the image is indexed later.
|
||||
4. ASSUMPTION: none. The enrichment pass is idempotent — if an image doc already has `image_context` set, it's skipped. The markdown doc's `image_context` is not cleared after enrichment (it's useful for retrieval display too).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_image_anchor_context.py` (task 06) — `_format_missing_image_refs` (single ref, multiple refs, empty list → empty string); the enrichment pass (markdown with missing refs → image doc enriched when found; image doc already enriched → skipped).
|
||||
- Integration: `tests/integration/test_importer.py` (extend) — seed a source with a markdown file referencing an image that doesn't exist yet → sync → the markdown doc has `image_context` set; then add the image file → sync again → the image doc is enriched with the context from the markdown doc. A markdown file referencing an image that exists → the image doc is enriched at index time (task 03 path).
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A markdown file referencing a non-existent image → the markdown doc's `image_context` stores the reference info.
|
||||
- [ ] When the image is later indexed (or re-synced), it picks up the context from the markdown doc.
|
||||
- [ ] A markdown file referencing an existing image → the image is enriched at index time (task 03 path), the markdown doc's `image_context` is NULL.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,38 @@
|
||||
# Task 05 — Retrieval display: show context line for enriched image docs
|
||||
|
||||
**Phase:** `125_image_anchor_context` · **Source:** `00_phase.md` — show context line for enriched image docs in the `<document>` block
|
||||
|
||||
## Objective
|
||||
When an enriched image doc (one with non-NULL `image_context`) appears in the suggested tier, the `<document>` block shows a short context line before the description — so the LLM knows why the image exists in the documentation.
|
||||
|
||||
## Work
|
||||
1. `app/rag/prompts.py` — extend the document block builder (`_document_block_body`, L438):
|
||||
- Currently it returns `doc.summary` (stripped). For image docs with `image_context`, prepend a context line:
|
||||
```python
|
||||
summary = (doc.summary or "").strip()
|
||||
if doc.is_image and doc.image_context:
|
||||
# Extract the first reference's alt text for a short context line
|
||||
# The image_context is formatted as "Referenced in <source>/<path>: ..."
|
||||
# Show: "Referenced in <path>: <alt_text>"
|
||||
context_line = _extract_context_line(doc.image_context)
|
||||
if context_line:
|
||||
summary = f"{context_line}\n\n{summary}"
|
||||
return summary
|
||||
```
|
||||
- `_extract_context_line(image_context: str) -> str | None`: parses the `image_context` string (set by task 04) and extracts a short context line:
|
||||
```python
|
||||
# image_context format: "Referenced in <source>/<markdown_path>:\n- [image.png] (line 42): \"Alt text here\""
|
||||
# Return: "Referenced in <markdown_path>: \"Alt text here\""
|
||||
```
|
||||
If parsing fails or the format is unexpected, return `None` (graceful degradation — the summary shows without the context line).
|
||||
2. ASSUMPTION: none. The context line is short (one line) so it doesn't bloat the prompt. Pre-phase image docs (NULL `image_context`) are byte-identical — the `if doc.is_image and doc.image_context` guard ensures this.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/unit/test_prompts.py` (extend) — `_document_block_body` for a non-image doc → unchanged (byte-identical to pre-phase); for an image doc with `image_context = None` → unchanged (byte-identical to pre-phase); for an image doc with `image_context` set → the summary includes the context line before the description; `_extract_context_line` (valid format → extracts the line; invalid format → returns None).
|
||||
- Integration: `tests/integration/test_chat_api.py` (extend) — a grounded turn whose suggested tier contains an enriched image doc → the `<documents>` section in the system prompt includes the context line; a non-enriched image doc → no context line.
|
||||
- Coverage: **>90%** on the touched modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] An enriched image doc in the suggested tier → the `<document>` block shows a context line before the description.
|
||||
- [ ] A non-enriched image doc → byte-identical to pre-phase (no context line).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,57 @@
|
||||
# Task 06 — Tests: unit, integration, and E2E for image-anchor context
|
||||
|
||||
**Phase:** `125_image_anchor_context` · **Source:** `00_phase.md` — the full test suite
|
||||
|
||||
## Objective
|
||||
Comprehensive tests across all new logic: the `extract_image_refs` function, the enrichment pipeline, the `describe_image` context extension, the retrieval display, and the end-to-end flow.
|
||||
|
||||
## Work
|
||||
1. `tests/unit/test_md_images.py` (new) — tests for `extract_image_refs`:
|
||||
- Single reference with alt text and surrounding paragraphs.
|
||||
- Multiple references on different lines, each with correct context windows.
|
||||
- References inside code fences (skipped).
|
||||
- References with no alt text (``).
|
||||
- References at start/end of file (context is partial).
|
||||
- References in headings.
|
||||
- Empty content, content with no references.
|
||||
- Verify `line_number` is 1-indexed and correct.
|
||||
- Edge cases: nested brackets in alt text, special characters in paths.
|
||||
2. `tests/unit/test_image_anchor_context.py` (new) — tests for the enrichment pipeline:
|
||||
- `describe_image` with `context=None` → same behavior as pre-phase.
|
||||
- `describe_image` with `context="..."` → the prompt includes the context prefix.
|
||||
- `_extract_context_line` (valid format → extracts the line; invalid format → returns None).
|
||||
- `_document_block_body` for enriched vs non-enriched image docs.
|
||||
- `_format_missing_image_refs` (single ref, multiple refs, empty list).
|
||||
- The enrichment pass (markdown with missing refs → image enriched; image already enriched → skipped).
|
||||
3. `tests/integration/test_importer.py` (extend) — integration tests:
|
||||
- Seed a source with a markdown file referencing an image → the image doc has `image_context` set.
|
||||
- A standalone image (no markdown reference) → `image_context` is NULL.
|
||||
- A markdown file referencing a non-existent image → the markdown doc stores the reference info.
|
||||
- Re-sync after adding the missing image → the image doc is enriched.
|
||||
- Verify that the enriched image doc's chunks are embedded from the enriched content.
|
||||
4. `tests/integration/test_chat_api.py` (extend) — integration tests:
|
||||
- A grounded turn whose suggested tier contains an enriched image doc → the done frame's system prompt includes the context line.
|
||||
- A non-enriched image doc → no context line (byte-identical to pre-phase).
|
||||
5. `tests/e2e/test_image_anchor_context.py` (new) — isolated E2E suite:
|
||||
- Module apps: `BOR_IMAGES=true` for this suite's app instance.
|
||||
- KB fixture: upload an archive containing a markdown file that references an image within the same archive (e.g., `docs/guide.md` references `diagram.png`).
|
||||
- Sync → the image doc is enriched with the markdown context.
|
||||
- Ask a question that requires the context to answer correctly (the mock LLM should use the context line to give a better answer).
|
||||
- Verify the `<document>` block in the prompt includes the context line (via the saved chat record).
|
||||
- A standalone image (no markdown reference) → the answer is byte-identical to pre-phase.
|
||||
- Run in isolation: `uv run pytest tests/e2e/test_image_anchor_context.py -v --no-cov`.
|
||||
6. Run the full test suite: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
|
||||
## Testing & Quality
|
||||
- All new logic is covered by unit tests (the `extract_image_refs` function, the enrichment pipeline, the context extraction/display).
|
||||
- Integration tests cover the full import + chat flow with seeded data.
|
||||
- E2E tests cover the real user path: upload → sync → ask → verify enriched answer.
|
||||
- The mock LLM should be configured to acknowledge the context line in its answer (extend the mock's prompt parser to recognize the context line format).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/unit/test_md_images.py -v --no-cov` passes.
|
||||
- [ ] `uv run pytest tests/unit/test_image_anchor_context.py -v --no-cov` passes.
|
||||
- [ ] `uv run pytest tests/integration/test_importer.py -v --no-cov` passes (extended).
|
||||
- [ ] `uv run pytest tests/e2e/test_image_anchor_context.py -v --no-cov` passes in isolation.
|
||||
- [ ] `uv run pytest` green; coverage TOTAL >90%; `ruff` + `pyright` clean.
|
||||
- [ ] No regression in existing E2E suites (run `test_image_documents.py` and `test_answer_images.py` to verify).
|
||||
Reference in New Issue
Block a user