.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. .. SPDX-License-Identifier: Apache-2.0 .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://www.apache.org/licenses/LICENSE-2.0 .. .. Unless required by applicable law or agreed to in writing, software .. distributed under the License is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. Visual Search ============= **Search mode: frame-level matching of a text query or an uploaded image against individual video frames.** Visual search retrieves clips by matching a text query or an uploaded image against individual video frames using SigLIP2 embeddings. Unlike semantic search, which represents each clip as a single video-level embedding, visual search operates at the frame level: a clip is returned if any single frame (or detected region within a frame) matches the query. This makes it particularly effective for finding specific objects, signs, or scene elements that appear briefly or only in part of a clip. **Reach for it when** the target is visually distinctive and identifiable from a still frame, such as a stop sign, a construction cone, or a wet road surface, and especially when it appears only briefly. You can also search by uploading a reference image instead of describing it. **Reach for something else when** the concept is about the clip as a whole unfolding over time rather than any single frame, where :doc:`semantic_search` matches the video-level representation. .. figure:: /_static/images/visual_search.png :alt: The Visual Search by Text field holding stop sign, with each result card showing its CLIP score for that phrase Visual search for ``stop sign``. Each clip is scored by how well a region within it matches the phrase. How It Works ------------ **Offline indexing.** For each clip, 8 frames are sampled uniformly across its duration. Each frame is processed in two ways: the full-resolution image is kept as one view, and **Florence-2** (``microsoft/Florence-2-base``) runs object detection (````) on the frame to propose up to 8 regions of interest (minimum 32 px per side, expanded by 10 %). Each detected region is cropped from the frame and treated as an additional view. All views -- full frames and region crops -- are encoded by **SigLIP2** (``google/siglip2-base-patch16-224``) into 768-dimensional vectors, L2-normalised, and stored in a FAISS inner-product index (default ``IVF4096,PQ64x8``). Because the number of detected regions varies per frame, the total number of indexed views per clip is variable rather than fixed. **At query time.** The query text is wrapped in a prompt template (default ``"a photo of {text}"``) and encoded by the same SigLIP2 model into a 768-d vector in the shared image-text space. FAISS returns the views whose visual content is nearest to it by cosine similarity. When multiple views from the same clip are retrieved, only the highest-scoring one is kept, so each clip appears at most once in the results. Clips are ranked by that best view score. Image Search ------------ .. figure:: /_static/images/visual_search_image.png :alt: An uploaded stop-sign image in the Visual Search by Image field, returning clips whose scenes contain stop signs Searching for visually similar clips by uploading `this image `_. Instead of typing a description, you can upload an image to use as the query. Click **Visual Search by Image** below the text query box and select a local file. The image is encoded by the same SigLIP2 model into a 768-dimensional vector and matched against the precomputed view index in the same way as a text query: each view's embedding is compared to the image embedding by cosine similarity, and clips are ranked by their highest-scoring view. Text-to-video and image-to-video search can be used together: when both are active, results are ranked by whichever score is higher per clip, with the text-to-video score taking priority when sorting. Query Rewriting --------------- .. figure:: /_static/images/visual_search_rewrite.png :alt: An Include in search box beneath the Visual Search by Text field listing four checked rewrites of stop sign Query rewriting expands ``stop sign`` into alternative phrasings, each shown as a checkable tag beneath the search field. Clicking **Rewrite** sends the text query to a language model that generates short alternative phrasings of the same visual concept. For example, "school zone sign" might expand to "speed limit sign", "crosswalk ahead sign", and "children crossing sign". The system runs a separate SigLIP2 search for each rewritten variant, deduplicates the results, and keeps the highest score per clip. Each rewrite appears as a tag in the UI; removing a tag excludes that variant from the search. Query rewriting applies to text queries only and is unavailable when image search is active. It is always enabled: when an NVIDIA or OpenAI API key is configured the rewrite is served by that provider, otherwise it falls back to a local LLM, so no key is required.