.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
.. SPDX-License-Identifier: Apache-2.0
..
.. Licensed under the Apache License, Version 2.0 (the "License");
.. you may not use this file except in compliance with the License.
.. You may obtain a copy of the License at
..
.. http://www.apache.org/licenses/LICENSE-2.0
..
.. Unless required by applicable law or agreed to in writing, software
.. distributed under the License is distributed on an "AS IS" BASIS,
.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.. See the License for the specific language governing permissions and
.. limitations under the License.

Caption Search
==============

**Search mode: keyword and phrase matching over the text captions attached to
each clip.**

Caption search matches the query against the captions associated with each
clip. All captions for a clip are searched together, so a match on any caption
surfaces the clip.

**Reach for it when** you can predict the words a caption will use, such as a
specific object, sign, manoeuvre, or road condition, and you want exact, fast
matching across the whole corpus with results you can verify by eye.

**Reach for something else when** the phrasing varies from clip to clip
(:doc:`caption_embed_search`), the concept may never have been captioned at all
(:doc:`semantic_search`), or the thing you want is visible in a single frame
rather than described in words (:doc:`visual_search`).

.. figure:: /_static/images/caption_search_crosswalk.png
   :alt: The Wheel interface with crosswalk typed into the Caption Search field and the matching clips listed below

   Caption search results for ``crosswalk``. The active query appears as a
   removable chip above the results.

Captions are indexed upfront using Full-Text Search (FTS). At search time
the index is looked up directly, making this fast even across hundreds of
millions of captions.

When no other ranked mode is active, results are returned in BM25 order:
rare terms that appear frequently in a clip's captions score more highly
than common terms. When combined with a ranked mode such as semantic or
classifier search, caption search acts as a filter and does not contribute
a ranking score of its own.

Query Behavior
--------------

All queries are normalised before being sent to the index:

* **Quoted phrases**: wrapping terms in double quotes, e.g. ``"heavy rain"``,
  matches clips whose captions contain that exact phrase in that order.
* **Unquoted multi-word input**: consecutive unquoted terms are automatically
  grouped into a quoted phrase. A query like ``heavy rain`` is treated as
  ``"heavy rain"``, not as two independent terms. Hyphens are replaced with
  spaces before this step, so ``stop-sign`` is treated as ``"stop sign"``.
* **Single terms**: a bare word like ``crosswalk`` matches clips whose captions
  contain that exact token.

.. figure:: /_static/images/caption_search_keyword_matching.png
   :alt: An expanded clip caption with every occurrence of the word crosswalk highlighted in yellow

   Matched query terms are highlighted directly in the captions of returned
   clips, making it easy to verify why each result was retrieved.

Boolean operators ``AND``, ``OR``, and ``NOT`` are supported. For example,
``crosswalk AND rain`` returns clips whose captions mention both, while
``rain NOT highway`` excludes clips mentioning highways.

.. figure:: /_static/images/caption_search_keyword_matching_not.png
   :alt: An expanded caption for the query rain NOT highway, with rain highlighted and no highlight for highway

   Retrieved results of caption search with query ``rain NOT highway``, i.e.
   find clips whose captions contain the word ``rain`` but not the word
   ``highway``.

Query Rewriting
---------------

.. figure:: /_static/images/caption_search_crosswalk_rewrite.png
   :alt: An Include in search box beneath the Caption Search field listing five checked rewrite variants of crosswalk

   Query rewriting expands ``crosswalk`` into five related phrases, each shown
   as a checkable tag beneath the search field.

Clicking **Rewrite** sends the query to a language model that generates
semantically related short phrases, the kind of phrasing that would appear
in a video description rather than a dictionary definition. Each variant
appears as a checkable tag in the UI; unchecking a tag removes it from the
search. Results from all active variants are merged so each clip appears
at most once.