.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. .. SPDX-License-Identifier: Apache-2.0 .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://www.apache.org/licenses/LICENSE-2.0 .. .. Unless required by applicable law or agreed to in writing, software .. distributed under the License is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. Deployment ========== This page explains how to deploy SIL-Wheel on a custom video dataset. Getting started requires running a set of pre-processing scripts to compute embeddings and other indexes for your data, then pointing the server at the resulting artifacts via a YAML configuration file. The server is modular: every datastore is optional, and the search modes that depend on a missing store are automatically disabled at startup. Quick Start on a Public Dataset ------------------------------- The fastest way to see what a working Wheel deployment looks like is to run the bundled `examples/getting-started-nuscenes/ `__ script. It downloads the public nuScenes mini split (10 scenes, no AWS account required), runs every preprocessing stage end-to-end, and starts a server with caption full-text search, Cosmos text-to-video, Qwen3 caption embeddings, Florence-2 + SigLIP2 visual search, and trajectory pattern + shape search all wired up. End-to-end takes roughly two hours on a single RTX 4090. A few features are intentionally skipped because they require additional artifacts: the BEV viewer, classifier and cluster search, the Arena, and VLM Judge stay disabled until their respective stores are populated. A second example, `examples/getting-started-physical-ai-autonomous-vehicles/ `__, runs the same stages against NVIDIA's Physical AI Autonomous Vehicles dataset. It streams the data from the Hugging Face Hub rather than downloading a split once, so ``--chunks`` and ``--max-clips`` control how much is processed and the run scales from a smoke test to a corpus of any size. The dataset is gated, so accept its licence on Hugging Face and log in first. Either script verifies the toolchain on a fresh machine before you bring up Wheel on your own data. nuScenes is the shorter path; Physical AI is the closer analogue to a real deployment, since it is large enough that you choose how much of it to ingest. Pre-processing -------------- Standing up Wheel on a custom dataset involves five offline stages, each backed by a script under ``scripts/`` in the wheel repository. The ``examples/getting-started-nuscenes/`` script linked above runs all five against the public nuScenes mini split and is the fastest way to see what each stage produces. .. note:: What follows is a summary. The full reference is `docs/data-preparation.md `__ in the wheel repository: exact S3 commands, schema migrations, dataset registration, country codes, FAISS index update snippets, and the supported input formats for each source. Read it before preparing real data; the five stages below are the map, not the territory. 1. **Process videos** (``scripts/prepare_data.py``). Downscales raw clips, re-encodes to H.264/AAC with faststart, and shards work via ``--n_processes`` / ``--process_id``. Inputs are a list of ``.mp4`` or ``.tar`` paths, local or S3. 2. **Upload to S3 and register clips.** ``aws s3 sync`` (or ``s5cmd``) the processed videos to the configured bucket, then upsert each clip's ``clip_id → relative_s3_path`` into the ``video_paths`` and ``clips`` tables of the annotations SQLite, plus register the dataset's category and license in the ``datasets`` table. 3. **Compute embeddings.** Three independent extraction passes over the same video list: - ``scripts/extract_video_text_embeddings.py`` for Cosmos text-to-video embeddings (powers semantic search). - ``scripts/extract_florence2_sigclip_embeddings.py`` for Florence-2 region crops encoded with SigLIP2 (powers visual search). - ``scripts/extract_captions.py`` followed by ``scripts/extract_captions_embeddings.py`` to produce captions and then their Qwen3 embeddings (powers caption FTS and caption embedding search). Each step appends its shards to the corresponding FAISS index. 4. **Compute trajectories** (``scripts/extract_trajectory_stats.py``). Produces per-clip safetensors, which are then assembled into the FAISS indices for the full-length, 10-second, and 5-second windows. 5. **Point the YAML at it.** Copy ``config/wheel_launch_prod_server_config.yaml``, replace every ``*_dir`` and ``*_db`` path with your local equivalents, and launch the server (see :ref:`launching-the-server` below). .. _what-each-search-mode-needs: What Each Search Mode Needs ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Every mode is backed by its own artifacts, and a mode whose store is missing from the config is hidden at startup rather than failing at query time. That is usually what an empty or absent search mode means. .. list-table:: :header-rows: 1 :widths: 26 30 44 * - Search mode - Config key - Built by * - :doc:`search/caption_search` - ``captions_db`` - ``extract_captions.py`` * - :doc:`search/caption_embed_search` - ``caption_embed_store`` - ``extract_captions.py``, then ``extract_captions_embeddings.py`` * - :doc:`search/semantic_search` - ``cosmos_embed_store`` - ``extract_video_text_embeddings.py`` * - :doc:`search/visual_search` - ``visual_embed_store`` - ``extract_florence2_sigclip_embeddings.py`` * - :doc:`search/trajectory_search` - ``trajectory_store`` - ``extract_trajectory_stats.py`` * - :doc:`search/perception_search` - ``wm_store`` - A 3D object detector run offline; not shipped with Wheel * - :doc:`search/classifier_search` - ``classifier_search`` - Trained in the UI, or ``train_classifier.py`` * - :doc:`search/cluster_search` - ``cluster_search`` - Run from the UI, or ``cluster_clips_and_select.py`` * - :doc:`search/metadata_search` (labels, source, geography) - ``annotations_db`` - ``prepare_data.py``, then annotation in the UI * - :doc:`search/metadata_search` (numeric metrics) - ``predictions_store`` - Model predictions written externally Three non-search features follow the same rule: the BEV viewer needs ``bev_store`` (``index_available_bev_metrics_files.py``), the Arena needs ``arena_db``, and the chat assistant needs ``agent_url`` (see :doc:`docs_assistant`). Trajectory search is the one worth watching, because its coverage can be partial rather than absent. If some clips carry ego trajectories and others do not, the mode works but silently searches only the clips that have them. The **Has Ego Trajectory** filter in :doc:`search/metadata_search` shows which those are. Configuration ------------- The server is configured via a YAML file with two top-level sections: ``datastores`` and ``server``. .. code-block:: yaml datastores: annotations_db: /path/to/annotations.db captions_db: /path/to/captions.db users_db: /path/to/users.db arena_db: /path/to/arena.db # optional, enables the Arena trajectory_store: trajectory_dir: /path/to/trajectories/ cosmos_embed_store: embeddings_dir: /path/to/cosmos_embeddings/ index_spec: "IVF4096,PQ96x8" visual_embed_store: embeddings_dir: /path/to/visual_embeddings/ index_spec: "IVF4096,PQ64x8" # optional, this is the default caption_embed_store: embeddings_dir: /path/to/caption_embeddings/ index_spec: "IVF4096,PQ128x8" # optional, this is the default wm_store: data_file: /path/to/wm_stats.parquet predictions_store: predictions_dir: /path/to/predictions/ classifier_search: classifier_dir: /path/to/classifiers/ cluster_search: clustering_dir: /path/to/clustering/ bev_store: s3_bucket: processed_data metrics_index_dir: /path/to/bev_index/ server: bindto: "0.0.0.0:8000" debug: false llm_provider: "auto" # "auto" | "openai" | "local" vlm_provider: "auto" # "auto" | "openai" | "local" vlm_judge_workers: 20 # parallel workers for arena VLM judging agent_url: "http://agent-host:8765" # optional, enables docs chatbot bug_report: spreadsheet_id: "" credential_path: /path/to/service_account.json The optional keys above behave as follows. ``arena_db`` enables the Arena (see :doc:`evaluation/arena`); when absent, the Arena page is disabled. ``llm_provider`` selects the backend for query rewriting and ``vlm_provider`` the backend for arena VLM judging; both take the same values. ``auto`` picks the first provider whose credentials are present in the environment (``openai`` → ``local``), while ``openai`` and ``local`` force a specific provider. ``local`` runs through a local model so no API key is required. ``vlm_judge_workers`` controls how many requests the VLM judge dispatches in parallel. ``agent_url`` points at a running SIL-Wheel docs agent server and enables the in-browser onboarding assistant described below. Slack integration for bug reports is configured out-of-band through a ``~/.slack/config`` and ``~/.slack/credentials`` profile on the server host rather than via the YAML. A reference config is provided at ``config/wheel_launch_dev_server_config.yaml``. Installation ------------ **1. Create the conda environment** (from the repository root): .. code-block:: bash conda env create -f environment.yml conda activate wheel **2. Build C extensions and install the package**: .. code-block:: bash python setup.py build_ext --inplace pip install -e . **3. Install S3 access tools** (required for video streaming): .. code-block:: bash pip install awscli .. note:: Do **not** install ``flash-attn``. There are known compatibility issues with CUDA 13.0 and PyTorch 2.11. PyTorch's built-in attention is used instead. .. note:: PE-Core (the ``pe_core_*`` embedding types in ``extract_video_text_embeddings.py``) needs Meta's `perception_models `_ package. Install it **after** the conda environment is set up, and with ``--no-deps`` — otherwise its ``transformers`` pin clobbers the version this project depends on: .. code-block:: bash pip install --no-deps git+https://github.com/facebookresearch/perception_models.git Skip this step if you don't plan to extract or evaluate ``pe_core_*`` embeddings. .. _launching-the-server: Launching the Server -------------------- The server is designed to run on a single workstation with a GPU. The reference deployment uses a desktop with an RTX 4090. .. code-block:: bash python scripts/launch_server.py [--override KEY=VALUE ...] The ``--override`` flag accepts dot-notation keys to override any config value without editing the file: .. code-block:: bash python scripts/launch_server.py config.yaml --override server.bindto=0.0.0.0:9000 Usage Analytics --------------- Wheel includes a built-in analytics dashboard that tracks usage over time. It shows the number of active users, search queries performed, and other activity metrics, making it easy to understand how the system is being used across teams. .. figure:: /_static/images/dashboard.png :alt: A usage dashboard with total search and user tiles, a daily unique users trend, search volume by type, and zero-result rate by type The usage analytics dashboard on April 1st 2026: search totals and daily unique users, volume broken down by search type, and the zero-result rate for each, flagged healthy, moderate or concerning. Credentials ----------- **Caption embedding search** requires authenticating with Hugging Face to download Qwen3-Embedding-8B: .. code-block:: bash pip install --upgrade huggingface_hub hf auth login **Query rewriting** is always enabled. When ``llm_provider`` is set to ``auto``, the server tries providers in order: 1. ``OPENAI_API_KEY`` for the OpenAI provider 2. A local LLM fallback that requires no API key The Rewrite buttons stay visible regardless of which provider is selected. Set ``llm_provider`` explicitly in the YAML to bypass auto-detection. **User authentication** is always enabled and the ``users_db`` key is required. Create an initial admin user directly in the ``users_db`` SQLite database before the first launch.