AIC Multimodal Search: finding moments, not files.

The user is not searching for a video. They are searching for a moment inside a video.

Media archives are hard because intent can be visual, spoken, object-based, temporal, or auditory. “A man opens a red door after a dog runs past” is not just a text query. It is a sequence of visual events. “Someone says climate policy” is transcript search. “Find the same audio clip” is fingerprint search. A useful system must handle all of them.

The system uses different indexes because each modality fails differently.

AIC is a retrieval core sitting between media archives and timestamped answers.

One video becomes several synchronized search surfaces.

The ingestion process loads keyframe embeddings into Milvus, object detections into MongoDB, transcripts into Elasticsearch, and audio fingerprints from HLS segments via FFmpeg. The key engineering concern is alignment: every signal must map back to the same video/keyframe timeline.

The query fans out, then collapses back to timestamped keyframes.

The Flask search API accepts text queries, object filters, and audio/transcript constraints. Each path returns candidate keyframes. The system intersects result sets by `(video_id, keyframe_index)` and ranks by fused scores when visual retrieval participates.

Visual retrieval uses two model perspectives instead of trusting one embedding space.

CLIP and BEIT3 queries are encoded separately, searched against separate Milvus collections, normalized, merged by keyframe, and fused with equal weights. The result is a ranked list of candidate keyframes with `fused_score`.

Some queries describe a sequence of events, not a scene.

The `temporal_search` method runs each query independently, groups candidates by video, chooses an anchor query, then walks backward and forward to find neighboring keyframes inside `MAX_FRAME_GAP`. This turns separate top-k lists into a temporal chain.

Object filters make retrieval concrete when language is too fuzzy.

MongoDB stores object detections per keyframe. The object search path uses aggregation with `$filter`, `$size`, `$gte`, and `$lte` to enforce label, confidence, minimum instance, and maximum instance constraints.

Speech needs lexical search because exact words matter.

Transcript CSV rows are cleaned, aligned to closest keyframes by FPS, and indexed in Elasticsearch with a custom analyzer. Querying combines fuzzy match and phrase match so spoken terms can retrieve timestamped frames.

Audio identity is different from speech meaning.

The fingerprint path concatenates HLS `.ts` segments with FFmpeg, converts them to mono WAV, extracts audio hashes, and stores them in a fingerprint database. This supports matching sound patterns even when transcript text is unavailable or irrelevant.

Retrieval is only complete when the result becomes playable and submittable.

The app maps each keyframe hit to FPS, shot boundaries, HLS playback data, and evaluation submission payload. This is where raw search hits become product results.

Multimodal search fails when indexes drift or one modality dominates.

Useful observability should track not just server health, but modality contribution, recall@K, stage latency, empty intersections, timestamp errors, and index freshness.

The demo should prove that different query types hit the same timeline.

Replace placeholders with app captures: semantic query, object-filter query, transcript query, temporal sequence query, HLS playback, and evaluation submit flow.

The strongest video starts with an impossible search prompt and ends on the exact moment.

The main lesson: multimodal search is about alignment, not just embeddings.