​
ADR-0051: Digital Worker media semantic search via text conversion

​
Status

Proposed (2026-07-23 — planned future phase; builds on ADR-0050)

​
Tags

digital-worker, ai, search, embeddings, media, multimodal, asr, vision, local, multilingual

​
Decision

To make non-text media (voice notes, images, video, documents) semantically searchable, Digital Worker will convert each media type to text with local models, then embed that text through the existing pipeline (bge-m3 → dw_message_embeddings, ADR-0050). It will not adopt native multimodal embeddings (Strategy B) in this phase. Concretely:
  1. Media → text with local models.
    • Voice / audio → transcript via a local ASR (语音转文字) model (Whisper — whisper.cpp / faster-whisper). Ollama does not do audio, so ASR is a new local service.
    • Image → OCR + description via a local vision-language model on the existing Ollama host (llama3.2-vision / qwen2.5-vl / llava): read any text in the image and describe it.
    • Video → keyframes + audio. Sample keyframes → vision description; extract the audio track → ASR. (Highest cost; last to land.)
    • Documents/PDF → text extraction (OCR fallback for scans).
  2. Enrichment is asynchronous and off the ingest hot path. A media-enrichment pass — the same watermark sweep shape as ADR-0050 — finds media events, runs ASR/vision, and writes the derived text. Ingest stays synchronous and unchanged (upholds ADR-0050’s async principle).
  3. Derived text is stored separately from the original. A new dw_chat_events.derived_text column (or equivalent) holds the transcript/caption/OCR; the original media placeholder ([image], [audio]) and the raw-payload reference are preserved for provenance (媒体溯源). The existing embedding sweep then embeds the derived text as ordinary text — no change to dw_message_embeddings, the KNN, hybrid search, or retention.

​
Why

  • One multilingual vector space. DW’s value is zh/ms/rojak recall (ADR-0050). Text conversion keeps everything in one bge-m3 space, so a query in any language still finds media-derived content. Most multimodal embedding models are English-centric and would forfeit this.
  • Speech meaning is in the words. For a voice note, embedding the transcript beats a multimodal audio embedding — the semantic content is linguistic, not acoustic. ASR-then-text is the correct method here, not merely the cheap one.
  • Screenshots dominate work chat. Much shared imagery is text (screenshots); OCR via a vision model captures exactly what a user would search for.
  • Reuses everything already built. The watermark sweep, poison handling, halfvec(1024) storage, model-scoped KNN, hybrid FTS+semantic read path, and retention coupling all apply unchanged. The only genuinely new component is the ASR service.
  • Preserves ADR-0028/0050 posture: local, unbilled, private, additive, off-hot-path. No chat media leaves the deployment.
Rejected alternatives:
  • Native multimodal embeddings (Strategy B — CLIP / SigLIP / Jina-CLIP / ImageBind / nomic-embed-vision). Introduces a second vector space (a different model, incompatible with bge-m3 rows), is generally English-centric (loses rojak), and its only real advantage — image↔image or non-speech audio similarity — is not a Digital Worker need. Deferred; revisit only if visual/ audio similarity search becomes a real requirement, at which point ADR-0050’s mandatory model filter already lets multimodal vectors coexist in a separate model-scoped table.
  • Paid multimodal APIs (Cohere Embed v4, Voyage multimodal). Per-item cost and sends internal chat media to a third party — same cost + privacy objection as paid text embeddings in ADR-0050.
  • Enrich inline at ingest. ASR/vision are heavy; running them on the hot path blocks ingest and re-introduces the exact cost/latency ADR-0050 removed. Rejected — async enrichment sweep.
  • Embed the [image]/[audio] placeholder as-is (today’s behavior). Semantically useless; a media-only window is skipped by the substance filter and is invisible to search. This ADR exists to replace that.

​
Consequences

  • New local dependency: an ASR service (Whisper). Vision reuses the existing Ollama host (add model pulls). Both are heavier per item than text embedding, so the enrichment sweep is watermark-paced and runs off-peak, exactly like the embedding sweep.
  • Schema: a derived_text column (+ optional media-type/confidence metadata) on dw_chat_events; no change to dw_message_embeddings.
  • Provenance + retention: the raw-payload reference is kept so the UI can show the source media; derived text dies with its source event under the same retention purge (ADR-0050 coupling) — it must not outlive event retention.
  • Quality gating: low-confidence transcriptions/captions are gated the same way as substance; deterministic enrichment failures reuse the poison-window pattern.
  • Phased delivery (highest value first): (1) voice notes, (2) images (OCR + caption), (3) video + documents.
  • bge-m3 / halfvec(1024) unchanged — media-derived text is just more text.

​
Rules for agents

  • Make media searchable by converting it to text with local models, then embedding that text through the existing bge-m3 pipeline — do not add a second (multimodal) vector space unless visual/audio similarity is an explicit requirement.
  • Run enrichment asynchronously, never on the ingest hot path (reuse the watermark-sweep + poison mechanics from ADR-0050).
  • Store derived text separately from the original media placeholder; keep the raw-payload reference for provenance.
  • Transcribe speech to text for voice notes — do not reach for audio embeddings; the meaning is in the words, and the transcript stays in the one multilingual vector space.
  • Derived text is a regenerable cache like the embeddings and dies with its source event under the retention purge.
  • Keep it local and unbilled — ASR and vision run on local models, never through AccessGate.