.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. .. SPDX-License-Identifier: Apache-2.0 .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://www.apache.org/licenses/LICENSE-2.0 .. .. Unless required by applicable law or agreed to in writing, software .. distributed under the License is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. Power Features ============== This section covers workflows that combine multiple Wheel capabilities to accomplish tasks that would not be possible with any single feature alone. Classifier Bootstrapping ------------------------ Classifier bootstrapping is the primary workflow for labeling rare or hard-to-describe scenarios at scale. It combines search, manual annotation, classifier training, and auto-labeling into an iterative loop that progressively scales a small number of human judgements to a much larger corpus. **The loop:** 1. Run any search to retrieve an initial set of candidates. A caption or semantic search for a rough description of the scenario is a good starting point. 2. Manually annotate a small number of positive examples within the UI. A few dozen is usually sufficient to get a usable first classifier. Optionally annotate negatives to sharpen the decision boundary. 3. Open the **Classifier** panel, configure the training options (embedding type, number of negative samples, whether to include autolabels), and click **Train Classifier**. Training completes in seconds. 4. Switch to :doc:`search/classifier_search` and retrieve clips scored above a high-confidence threshold (e.g. ``p > 0.9``). Inspect the results to assess quality. 5. Use **Auto-label** to stamp the high-confidence results with the target label in one click. 6. Retrain the classifier with the expanded annotation set by checking **Train with autolabels**. The additional labeled examples shift the decision boundary and improve recall. 7. Repeat from step 4, progressively lowering the threshold or inspecting the boundary region (e.g. ``0.7 < p < 0.9``) to find more true positives. Each iteration expands the labeled set and improves the classifier, making the next retrieval more precise. After a few rounds, the classifier can reliably surface the target scenario across millions of clips with minimal additional human review. .. note:: The classifier can be trained on three embedding spaces: CosmosEmbed1 (video-level), Qwen3-Embedding-8B (caption-level), or Florence-2 / SigLIP2 (frame-level). Use CosmosEmbed1 for whole-clip scenarios, caption embeddings when the distinguishing factor is best expressed in language, and SigLIP2 when the target is a visually distinctive object, sign, or scene element that may appear only briefly. All three can be trained and used independently under the same label. Data Flywheel ------------- Hard-example mining identifies the clips that are most challenging for a model and uses them to build targeted evaluation slices, training subsets, or seeds for further data collection. The strategy is the same whether the goal is to stress-test a model or to drive the next round of curation: surface failure modes from evaluation, find more clips that exhibit them, and close the loop. **The workflow:** 1. Run a benchmark on a broad slice via the leaderboard to get per-clip model scores. 2. Inspect the leaderboard results to identify the clips with the lowest scores — these are the model's hardest cases. 3. Inspect the hard examples to identify whether they share a common scenario (e.g. a particular weather condition, intersection type, or object configuration). 4. Use search (caption, semantic, trajectory, perception-based, or a combination) to retrieve more clips covering that scenario from the existing dataset. 5. Curate the slice: remove false positives manually or with label filters, then auto-label the remainder. 6. Use the curated slice as a hard benchmark and as a seed for a classifier that can be applied to incoming data to flag new examples of the same failure mode automatically. This closes the loop between model evaluation and data curation: failures discovered in evaluation directly drive what gets annotated and collected next. The :doc:`agent` can automate steps 2–4 by combining leaderboard feedback with Wheel's retrieval modes in a single natural-language request. VLM Judge --------- VLM Judge provides two distinct workflows that use a vision-language model to score clip caption quality and validate retrieval quality directly from video content. Caption Scoring ^^^^^^^^^^^^^^^ Caption scoring evaluates how well a caption describes a clip. The model returns attribute-level scores (scene, action, road entities, temporal, and overall) together with short reasoning. .. figure:: /_static/images/vlm_judge_caption_scoring.png :alt: A time-segmented caption with an Evaluate Caption button, scored 9 for scene, action, road entities and temporal for an overall 8 out of 10, above the judge's written critique :width: 95% VLM Judge caption scoring flow in the annotation interface. **The workflow:** 1. Open a clip in the annotation interface and launch **VLM Judge**. 2. Provide or select the caption to evaluate. 3. Run caption scoring to get per-attribute quality scores plus reasoning. Caption scoring is useful for caption QA, model comparison, and iterative caption-prompt development. Search Validation ^^^^^^^^^^^^^^^^^ Search validation adds a model-based precision check on top of retrieval results. It verifies whether top-ranked clips actually satisfy the active query. .. figure:: /_static/images/vlm_judge_search_validation.png :alt: The VLM Judge dialog with a text query of crosswalk and Top K of 20, ready to check the top-ranked results of the current search :width: 95% VLM Judge search validation flow in the Advanced Search panel. **The workflow:** 1. Run either caption search, caption embedding search, or semantic text search to retrieve a ranked list. 2. Open **VLM Judge** in the **Auto-label** panel and select the query and modality to validate. 3. Set **Top K** to choose how many of the highest-ranked clips to validate. 4. Run validation. For each clip, VLM Judge inspects sampled frames and returns structured reasoning plus a binary query-match prediction. 5. Use validated positives for review or auto-labeling, and use negatives to refine the query before rerunning retrieval. This workflow is most useful when the retrieval query is broad or ambiguous: it quickly separates clearly relevant clips from borderline results so annotation effort focuses on high-value candidates. .. figure:: /_static/images/vlm_judge_search_validation_output.png :alt: A green Match verdict overlaid on each clip, with the judge citing the frames where the crosswalk markings appear :width: 95% VLM Judge reviewing whether the retrieved videos contain a crosswalk. Each clip receives a match or no-match verdict, allowing users to filter out false positives before annotating. .. note:: VLM-Judge is available when a VLM provider is configured and VLM dependencies are installed. With the default ``auto`` provider setting the server uses OpenAI when ``OPENAI_API_KEY`` is set and otherwise falls back to a local VLM that requires no API key. The UI reuses the active search query and ranked IDs, and caps validation to the available top-ranked results. Agent-Driven Hybrid Search -------------------------- The :doc:`SIL-Wheel Agent ` adds a natural-language interface on top of Wheel's composable search modes. Rather than configuring filters manually, users describe a scenario in plain English and the agent selects, combines, and executes the appropriate search modes automatically. For multi-strategy retrieval, the agent's built-in ``find_clips_for_scenario()`` method runs caption search, semantic search, and matching classifier filters in parallel, then merges results via reciprocal rank fusion. This surfaces clips that no single search mode would rank highly on its own — for example, clips that match a caption keyword *and* score well on a trained classifier *and* are semantically close to a text description. The agent can also chain retrieval with curation: starting from a seed set of clips, it expands the set through similarity search across visual and trajectory dimensions, then exports the result for annotation, training, or evaluation. See :doc:`agent` for setup instructions, the full list of supported search modes, and Python/CLI usage examples.