# Multimodal: image extraction + indexing for documents

**Status:** Spec, not started. Branch will be cut from `main` at commit `63d8538`.

**Goal:** When a user ingests a PDF / PPTX / DOCX that contains images
(charts, diagrams, photos, screenshots), the images become discoverable
via the existing wiki search + chat flow alongside the document's text.

**Non-goals (this round):**
- "Search by image" / image-to-image retrieval — deferred to Phase 5.
- Editing or annotating images post-ingest.
- OCR-only path (Tesseract). VLM caption is strictly more capable; if
  cost becomes the issue we'll add OCR as a fallback later.
- Replacing / changing the existing chunker, embedding API, search
  ranking, or RAG pipeline. This work strictly **adds** a vision step
  on top.

---

## Current state (audit, not assumption)

`src-tauri/src/commands/fs.rs::preprocess_file`:

| Format | Current behavior | Image handling |
|---|---|---|
| PDF | `pdfium_render` → `page.text().all()` | **Ignored.** Embedded images, scans, charts all dropped. |
| PPTX | unzip → parse `ppt/slides/slideN.xml` | **Ignored.** `ppt/media/*.png|jpg` already in the ZIP, just not read. |
| DOCX | unzip → parse `word/document.xml` | **Ignored.** `word/media/*` same as above. |
| XLSX/ODS | `calamine` → cell text | Ignored. |
| Standalone images (.png/.jpg/...) | Read as binary in `read_file`; preview UI shows them | **Do NOT enter the ingest pipeline.** Never become wiki pages. |

Whole TS chain (`text-chunker.ts`, `embedding.ts`, `search.ts`,
`chat-panel.tsx`) is text-only. LanceDB v2 schema field is
`chunk_text: Utf8` — no provision for image bytes or paths.

Dependencies already present we can lean on:
- `pdfium-render` 0.9 — supports `page.objects()` iteration, including
  `PdfPageObjectType::Image` extraction
- `zip` 2.x — direct access to PPTX/DOCX `media/` directories
- LLM provider abstraction in `llm-providers.ts` — every provider
  (OpenAI / Anthropic / Gemini / Claude Code CLI) supports
  vision-message input on its native wire; we just don't expose it

---

## Design: caption-first hybrid (Option C from planning)

Three rejected alternatives are documented at the bottom of this file
for posterity. The chosen path:

1. **Extract images** from PDF / PPTX / DOCX during preprocess
2. **Save originals** to `<project>/wiki/media/<source-slug>/<n>.<ext>`
3. **Caption with vision LLM** ("describe factually, include any text,
   chart axes, key visual elements; 2–4 sentences")
4. **Inject as markdown** `![<caption>](media/<source-slug>/<n>.png)`
   into the source content fed to the analysis / generation prompts —
   so the LLM that builds the wiki page can place these images
   contextually
5. **Captions are ordinary text** — they flow through the existing
   `chunkMarkdown` → `embedPage` → `vector_upsert_chunks` pipeline
   with zero changes
6. **chat-message renders the markdown image** — the existing
   `react-markdown` setup already does this; it just needs the path
   to resolve to the right place

### Why this design

- **No schema change** to LanceDB. Captions are text chunks. Search
  works without modification.
- **No retrieval-quality regression.** Existing text-only retrieval
  paths are untouched. The chunker just sees more text (the
  captions) which makes images cite-able by their semantic content.
- **User sees the actual image** in chat replies, not just a textual
  description.
- **Provider-agnostic.** Every LLM provider we support has a vision
  format; we abstract over them in `buildBody`.
- **Phased.** Each phase is independently shippable and reversible.

### What this design does NOT solve (and that's OK for v1)

- Retrieving "an image that LOOKS like X" (visual similarity) — needs
  multimodal embedding (Phase 5, deferred).
- Captions that miss subtle details (e.g. "the third bar is taller
  than the second") — limited by VLM quality. Pro-tier models help;
  Flash Lite captions will be shallow.
- Image dedup across files (same logo / icon appearing 50 times) —
  handled by a SHA-256 hash cache, see Phase 1 risks below.

---

## Implementation phases

### Phase 1: Rust-side image extraction

New commands in `src-tauri/src/commands/fs.rs` (or a new
`src-tauri/src/commands/extract_images.rs` if `fs.rs` is getting too
big — currently 1100+ lines, leaning toward new file).

Public API shape (Tauri commands, callable from TS):

```rust
#[derive(Serialize)]
struct ExtractedImage {
    /// 1-based image index within the document (for filename)
    index: u32,
    /// PNG / JPEG / etc., as a MIME type
    mime_type: String,
    /// Page (PDF) or slide (PPTX) the image came from. None for DOCX.
    page: Option<u32>,
    /// Pixel width / height — used to filter out logos / icons.
    width: u32,
    height: u32,
    /// Image bytes, base64-encoded for IPC.
    data_base64: String,
}

#[tauri::command]
async fn extract_pdf_images(path: String) -> Result<Vec<ExtractedImage>, String>

#[tauri::command]
async fn extract_office_images(path: String) -> Result<Vec<ExtractedImage>, String>
```

Implementation notes:
- **PDF**: iterate `doc.pages()` → `page.objects()` → filter
  `PdfPageObjectType::Image` → `as_image_object().get_raw_image()`
  → encode to PNG via `image` crate (already a transitive dep
  through pdfium-render).
- **PPTX/DOCX**: open as ZIP, iterate file names matching
  `^(ppt|word)/media/.*\.(png|jpe?g|gif|webp|bmp)$`, read bytes
  directly — already in their native format.
- **Size filter**: drop images smaller than 100×100 (configurable
  later). Saves VLM cost on logos / decorations / cropping
  artifacts. ~80% noise removal in practice for slide decks.
- **Memory**: extract images in a `for` loop, not `collect()` — a
  100-page PDF with 50 images is ~50 MB before base64 (~67 MB
  after). Streaming through a `Vec<ExtractedImage>` is OK for IPC
  but we should be defensive against a pathological 5000-image
  document — add a `max_images: 500` cap.

Tests (`src-tauri/src/commands/extract_images.rs::tests`):
- Synthetic PDF with 1 known image → extract returns 1 entry with
  expected dims and non-empty bytes.
- Real PPTX from `tests/fixtures/` with multiple slides containing
  images → counts and sizes match.
- DOCX with no images → returns `Ok([])`, not an error.
- Password-protected PDF → returns the same error string the text
  extractor returns (consistent UX).

### Phase 2: Vision-message support in LLM abstraction

`src/lib/llm-providers.ts`:

```ts
// New union — replaces the existing `content: string` on ChatMessage
export type ContentBlock =
  | { type: "text"; text: string }
  | { type: "image"; mediaType: string; dataBase64: string }

export interface ChatMessage {
  role: "system" | "user" | "assistant"
  // Backwards-compatible: providers that don't get an image keep
  // calling sites working with plain strings. Block-array form
  // unlocks vision input.
  content: string | ContentBlock[]
}
```

Each provider's `buildBody` learns to translate `ContentBlock[]`:
- **OpenAI**: `[{type:"text",...}, {type:"image_url",image_url:{url:"data:image/png;base64,..."}}]`
- **Anthropic**: `[{type:"text",...}, {type:"image",source:{type:"base64",media_type:"image/png",data:"..."}}]`
- **Gemini**: `parts:[{text:"..."},{inline_data:{mime_type:"image/png",data:"..."}}]`
- **Claude Code CLI**: already takes content blocks (PR #61), just
  add `image` block type passthrough.
- **Ollama**: `messages[].images: [base64]` (separate field, not
  inline blocks). Conditional on the model — only `llava`,
  `qwen2.5-vl`, etc. accept it.

Existing test files (`llm-providers.test.ts`,
`__tests__/claude-cli-transport.test.ts`) need vision cases added.

### Phase 3: Captioning helper + ingest integration

`src/lib/vision-caption.ts` (new):

```ts
export async function captionImage(
  imageBase64: string,
  mediaType: string,
  llmConfig: LlmConfig,
  signal?: AbortSignal,
): Promise<string>
```

Implementation: build a `streamChat` call with a single user message
whose content is `[{type:"text",text:CAPTION_PROMPT},{type:"image",...}]`,
collect all tokens, return the joined string.

Caption prompt (pinned, factual, no markdown):

> Describe this image factually for a knowledge-base index. Include:
> any visible text verbatim, chart axes and values, diagram structure
> (boxes/arrows/labels), key visual elements. Do NOT speculate or
> editorialize. 2 to 4 sentences. Output plain text only — no
> markdown, no preamble.

`src/lib/ingest.ts` integration:

After `preprocess_file` returns text, BEFORE the analysis stage:

```ts
const images = await invoke('extract_pdf_images' or 'extract_office_images', { path })
const captioned = []
for (const img of images) {
  const relPath = `wiki/media/${slug}/img-${img.index}.${ext}`
  await writeFile(`${pp}/${relPath}`, base64ToBytes(img.data_base64))
  const caption = await captionImage(img.data_base64, img.mime_type, llmConfig, signal)
  captioned.push({ relPath, caption, page: img.page })
}

// Inject into sourceContent so the LLM sees them in context
const imageSection = captioned.length > 0
  ? '\n\n## Embedded Images\n\n' +
    captioned.map(c =>
      c.page
        ? `**[Page ${c.page}]** ![${c.caption}](${c.relPath})`
        : `![${c.caption}](${c.relPath})`
    ).join('\n\n')
  : ''
const enrichedSource = sourceContent + imageSection
// ... rest of autoIngest uses enrichedSource
```

Per-image cache keyed by SHA-256 of image bytes — same logo across
50 PDFs = 1 caption call, not 50. Cache lives in
`<project>/.llm-wiki/image-caption-cache.json` mapping
`hash → caption` (and image dimensions, mime, optionally the cached
file path so we deduplicate file storage too).

### Phase 4: Settings toggle + cost guardrails

`src/components/settings/sections/embedding-section.tsx` (or a new
"Multimodal" section if it grows): add a toggle.

```
☐ Index images from documents (uses extra LLM credits)
   Each image is captioned with a vision model. A 100-page paper
   with 30 images = 30 vision calls per ingest.
   Max images per document: [500]
   Skip images smaller than: [100]px on either side
```

Stored in `useWikiStore.embeddingConfig` (or a sibling
`multimodalConfig` if we want to keep them separate). Read by
`autoIngest` to decide whether to run Phase 1 + 3 at all.

**Default off.** Users opt in. README / changelog notes the cost
implication clearly.

### Phase 5 (deferred, NOT this round): multimodal embedding

Add a parallel embedding path that hits a multimodal endpoint
(`/v1/embeddings` with image input — supported by Voyage Multimodal,
Jina CLIP v2, some local CLIP servers). Store image-vector alongside
text chunk-vector in LanceDB (either same table with a `kind` field,
or a sibling `wiki_images` table).

This unlocks "find an image that looks like X" but is **strictly
additive** — caption-based retrieval keeps working as-is.

Requires user to have a multimodal embedding endpoint, which their
current LM Studio `qwen3-embedding-0.6b` is NOT.

---

## Open questions (resolve before / during Phase 1)

1. **Provider matrix**: which providers should the vision toggle
   actually enable? OpenAI / Anthropic / Gemini / Claude Code CLI all
   work. Ollama needs a vision-capable model (must check `cfg.model`
   against a known list). Custom endpoint depends on user's setup.
   MiniMax — uncertain, needs probe. **Tentative answer**: silently
   skip vision step on providers that don't support it; show a banner
   in Settings.

2. **Image size threshold**: 100×100 vs 80×80 vs 5KB byte threshold.
   Small images are usually icons / decorations. **Tentative**:
   100×100 default, exposed in Settings.

3. **Dedup strategy**: SHA-256 hash of image bytes → cache caption
   for that hash project-wide. **Tentative**: yes, default on.
   Cache invalidation tied to caption-prompt version.

4. **Per-document VLM cap**: a 500-page slide deck with 1500 images
   could blow up costs unnoticed. **Tentative**: hard cap at 500
   images per document, configurable. Beyond that, surface a
   warning in the activity panel and skip.

5. **What if the VLM call fails / times out?** Caption-less image
   should still be saved to disk and embedded as `![image](path)`
   without a caption — it's at least visible to the user, just not
   searchable by content. Soft failure, not hard.

6. **Standalone .png / .jpg imports**: do we treat them as
   single-image "documents" and run them through the caption path?
   **Tentative**: yes, but as a follow-up after Phase 1–4 land for
   embedded images.

7. **Image sub-dir naming**: `wiki/media/<source-slug>/` or flat
   `wiki/media/<slug>-<n>.<ext>`? Subdirs are cleaner; conflicts
   resolved by source-delete cascade automatically. **Tentative**:
   subdirs.

---

## Risks

- **Cost**: Phase 3 is the expensive step. Mitigated by Phase 4
  toggle (default off) + dedup cache + per-doc cap.
- **Caption quality**: Flash Lite produces near-useless 1-sentence
  captions. Document this in Settings hint; recommend Sonnet+ for
  multimodal.
- **Performance**: each ingest now does N additional LLM calls in
  series. For a 30-image PDF, that's 30 × ~3s = 90s extra latency.
  We can parallelize the caption calls with `Promise.all` (the
  caption mutex doesn't apply — they're independent).
- **PDF extraction quality**: pdfium's image extraction returns the
  raw embedded image; for vector graphics (which PDFs sometimes
  use for charts) this fails — those are paths/text, not Image
  objects. We'd miss them. **Mitigation**: render the entire page
  to a PNG as fallback when no Image objects found AND the page
  has structural complexity. Defer to Phase 1.5 if Phase 1
  results are weak.

---

## Testing strategy

Per phase, in priority order:

**Phase 1 (Rust extraction):**
- Unit tests with synthetic + real fixtures
- Test on a known-good PDF (e.g. an arxiv paper) — verify image
  count matches manual count

**Phase 2 (vision message format):**
- Per-provider unit tests: assert correct wire format for each
- Mock-server test that the bytes-on-wire match each provider's
  documented schema

**Phase 3 (captioning + ingest):**
- Real-LLM test (gated by `RUN_LLM_TESTS=1`): pass a known image,
  verify caption is non-trivial and contains expected keywords
- Integration test: full autoIngest on a small fixture PDF with 2
  known images → assert wiki/media/ has the files + the generated
  page references them in markdown

**Phase 4 (toggle):**
- UI smoke test (manual)
- Unit test: when toggle is off, extract_*_images is never called

---

## Rejected alternatives (for posterity)

- **Pure VLM caption** (Option A): same as our chosen path BUT
  without saving the original image. User loses ability to see the
  image in retrieval. Rejected — UX regression.
- **Pure multimodal embedding** (Option B, no caption): no LLM cost
  at index time, true semantic image retrieval. Rejected because
  user's current embedding endpoint is text-only, AND we lose the
  ability to feed image content into LLM context (no caption text).
- **OCR-only**: useless for non-textual images (charts, photos).
  Rejected for v1; could be added as a fallback for VLM failures.

---

## Branch + delivery plan

- Cut branch `feat/multimodal-images` from `main` @ `63d8538`
- Phase 1 → 1 commit, ~3-4 days work
- Phase 2 → 1 commit, ~2 days
- Phase 3 → 2 commits (caption helper + ingest integration), ~3 days
- Phase 4 → 1 commit, ~1 day
- Each commit independently runnable + tested. Merge phases into
  branch as they land. Final merge to main as one big feature, OR
  as 4 separate PRs depending on review preference.

Total estimate: ~10 days of focused work.
