.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
.. SPDX-License-Identifier: Apache-2.0
..
.. Licensed under the Apache License, Version 2.0 (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.

Semantic Search
===============

**Search mode: meaning-based matching against video embeddings of the clip
itself, rather than any text about it.**

Semantic search retrieves clips using video-level embeddings. Unlike
caption-based search, it matches against the clip representation itself rather
than the associated text, so it can return relevant clips even when the
captions use different wording or omit the concept entirely.
It supports two modes: text-to-video, where a text
query is matched against clip embeddings, and video-to-video, where a
reference clip is used as the query. Both modes can be active simultaneously.

**Reach for it when** the property you care about is visual and may never have
been written into a caption. Use video-to-video when you have a strong
reference example but cannot describe it precisely in words, or when you want
to grow a small seed set into a larger slice of similar clips.

**Reach for something else when** the concept is reliably captured in text,
where :doc:`caption_embed_search` searches the descriptions rather than the
pixels, or when the target is a specific object visible in a single frame,
where :doc:`visual_search` matches frame by frame.

.. figure:: /_static/images/semantic_search.png
   :alt: The Text Search field holding car stopping behind a school bus, with each result showing a school bus ahead and a semantic text score

   Semantic text-to-video search for ``a car stopping behind a school bus``.
   Results are clips whose video content matches the description regardless
   of how they are captioned.

How It Works
------------

**Offline indexing.** Each clip is encoded offline into a 768-dimensional embedding using
**Cosmos-Embed1**, which maps video and text into a shared embedding space.
A text description and a visually matching clip end up
close to each other in this space regardless of how the clip is captioned.
Vectors are L2-normalised and stored in a FAISS IVF index (one entry per
clip), so nearest-neighbour search is fast regardless of dataset size.

**Text-to-video.** The query text is encoded by the same CosmosEmbed1 model
into a vector in the shared space. FAISS returns the clips whose video
content is nearest to it by cosine similarity. A query like "pedestrian holding
an umbrella" can retrieve matching clips even when that phrase does not appear
in the captions.

**Video-to-video.** A reference clip's embedding is looked up directly from
the index and used as the query vector. FAISS returns the clips that are
closest to it. To use this mode, click **Semantic Search** on any clip card
in the UI and the selected clip becomes the query automatically. The reference
clip itself always appears first in the results, followed by the most similar
clips in descending order.

.. figure:: /_static/images/semantic_search_video_to_video.png
   :alt: The Video Search field holding a reference clip ID, returning narrow parked-up street scenes each with a semantic video score

   Semantic video-to-video search using a reference clip. Results are
   visually and semantically similar clips retrieved directly from the
   reference embedding.

Text-to-video and video-to-video can be used together. Each produces its own
score for a clip. When both are active, the ranking follows the global
search-mode priority described in :doc:`index`.

.. figure:: /_static/images/semantic_search_combined.png
   :alt: Two active chips, semantic text search for crosswalk and semantic video search on a reference clip, with both scores on every result card

   Text-to-video and video-to-video semantic search active simultaneously.
   Each clip is scored against both the text query and the reference clip
   embedding.

Query Rewriting
---------------

.. figure:: /_static/images/semantic_search_rewrite.png
   :alt: An Include in search box beneath the Text Search field listing five checked rewrites of officer in construction zone

   Query rewriting expands the text query into alternative phrasings, each
   shown as a checkable tag in the UI.

Query rewriting applies to text-to-video queries only. Video-to-video mode
uses a clip embedding directly and cannot be rewritten. Clicking **Rewrite**
sends the query to a language model that generates alternative phrasings of
the same concept: synonyms, related terms, and descriptions using different
vocabulary. The system runs a separate search for each rewritten variant,
deduplicates the results, and keeps the best score for each clip.