.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
.. SPDX-License-Identifier: Apache-2.0
..
.. Licensed under the Apache License, Version 2.0 (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.

Caption Embedding Search
========================

**Search mode: meaning-based matching over embeddings of the clip captions.**

Caption embedding search matches the query against dense embeddings of the
captions associated with each clip.  Unlike :doc:`caption_search`, which
requires the query words to appear verbatim, this mode finds clips whose
captions express the same concept in different words, making it effective for
broad or paraphrase-heavy queries. In practice, this retrieves clips based on
semantic similarity, so relevant results can be found even when the query words
do not appear exactly in the caption text.

**Reach for it when** the concept is reliably described in captions but the
wording varies from clip to clip. A query for "pedestrian crossing with
umbrella" also surfaces clips captioned "person walking in rain" or "woman with
umbrella at crosswalk"; each clip is ranked by its best-matching caption.

**Reach for something else when** the wording is known and consistent, where
:doc:`caption_search` is faster and more predictable, or when the property is
visual and may never have been written down at all, where
:doc:`semantic_search` works over the video itself.


.. figure:: /_static/images/caption_embed_search.png
   :alt: The Caption Embedding Search field holding pedestrian crossing with umbrella, with each result card showing its caption embed score

   Caption embedding search for ``pedestrian crossing with umbrella``. Results
   include clips whose captions describe the same scene in different words,
   without the query phrase appearing verbatim.

How It Works
------------

**Offline indexing.** Every caption is encoded into a dense vector using
**Qwen3-Embedding-8B** (via ``sentence-transformers``), a text embedding 
model optimised for semantic similarity. Vectors are L2-normalised and added
to a FAISS IVF index. Since a clip typically has
captions from multiple sources or time windows, a clip can in turn have
multiple entries in the index.

**At query time.** The query text is encoded by the same model into a
vector in the same space. FAISS returns the nearest caption vectors by
cosine similarity. If multiple captions from the same clip are retrieved, only
the highest-scoring match is kept, so each clip appears at most once in the
results.

Query Rewriting
---------------

.. figure:: /_static/images/caption_embed_search_rewrite.png
   :alt: An Include in search box beneath the Caption Embedding Search field listing five checked umbrella-related rewrite variants

   Query rewriting expands ``pedestrian crossing with umbrella`` into related
   phrasings, broadening recall to clips captioned with any of them.

Clicking **Rewrite** sends the query to a language model that generates
alternative phrasings of the same concept: synonyms, related terms, and
descriptions using different vocabulary that might match how captions were
actually written (e.g. "pedestrian crossing with umbrella" might expand to
"person walking in rain", "woman with umbrella at crosswalk", "rainy
intersection pedestrian"). The search runs for each variant in parallel and
results are merged. Each clip appears exactly once, ranked by its best score
across all active queries. This broadens the pool of retrieved results, while
the final ranking still depends on the strongest match for each clip.