Caption Embedding Search

Search mode: meaning-based matching over embeddings of the clip captions.

Caption embedding search matches the query against dense embeddings of the captions associated with each clip. Unlike Caption Search, which requires the query words to appear verbatim, this mode finds clips whose captions express the same concept in different words, making it effective for broad or paraphrase-heavy queries. In practice, this retrieves clips based on semantic similarity, so relevant results can be found even when the query words do not appear exactly in the caption text.

Reach for it when the concept is reliably described in captions but the wording varies from clip to clip. A query for “pedestrian crossing with umbrella” also surfaces clips captioned “person walking in rain” or “woman with umbrella at crosswalk”; each clip is ranked by its best-matching caption.

Reach for something else when the wording is known and consistent, where Caption Search is faster and more predictable, or when the property is visual and may never have been written down at all, where Semantic Search works over the video itself.

Caption embedding search for pedestrian crossing with umbrella. Results include clips whose captions describe the same scene in different words, without the query phrase appearing verbatim.

How It Works

Offline indexing. Every caption is encoded into a dense vector using Qwen3-Embedding-8B (via sentence-transformers), a text embedding model optimised for semantic similarity. Vectors are L2-normalised and added to a FAISS IVF index. Since a clip typically has captions from multiple sources or time windows, a clip can in turn have multiple entries in the index.

At query time. The query text is encoded by the same model into a vector in the same space. FAISS returns the nearest caption vectors by cosine similarity. If multiple captions from the same clip are retrieved, only the highest-scoring match is kept, so each clip appears at most once in the results.

Query Rewriting

Query rewriting expands pedestrian crossing with umbrella into related phrasings, broadening recall to clips captioned with any of them.

Clicking Rewrite sends the query to a language model that generates alternative phrasings of the same concept: synonyms, related terms, and descriptions using different vocabulary that might match how captions were actually written (e.g. “pedestrian crossing with umbrella” might expand to “person walking in rain”, “woman with umbrella at crosswalk”, “rainy intersection pedestrian”). The search runs for each variant in parallel and results are merged. Each clip appears exactly once, ranked by its best score across all active queries. This broadens the pool of retrieved results, while the final ranking still depends on the strongest match for each clip.