.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. .. SPDX-License-Identifier: Apache-2.0 .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://www.apache.org/licenses/LICENSE-2.0 .. .. Unless required by applicable law or agreed to in writing, software .. distributed under the License is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. Architecture ============ System Overview --------------- Wheel employs a central HTTP server that manages concurrent access to specialized data stores, where different stores handle different modalities such as trajectories, captions, and labels. It supports multiple concurrent users through a thread-per-request model. A separate WebSocket connection handles real-time viewer events. Compute-intensive tasks such as classifier training and clustering run as managed subprocesses. Offline preprocessing populates the specialized stores before the server starts. At runtime the server is the single access point: every client sends its requests through it, and the stores are never accessed directly. .. figure:: /_static/images/sil_wheel_system_architecture.png :alt: A user reaching the SIL-Wheel HTTP server through either the browser UI or programmatic access, with the server fanning out to six stores: video, FAISS indices, SQLite, metrics, scenario scores, and full-text search :width: 90% :align: center SIL-Wheel system architecture. The browser UI and the Python clients have access to the same data and the same search modes, because both go through a single HTTP server. Core Design Principles ---------------------- * **Clip-centric representation**: The clip is the common unit across search, annotation, curation, and evaluation. All retrieval methods resolve to clip IDs, so results from any search mode can be used directly for annotation, export, classifier training, or evaluation. * **Specialized stores**: Different modalities require different backends. Full-text retrieval, vector similarity search, annotation storage, structured filtering, and evaluation state are handled by separate stores chosen for their specific access pattern. * **Composable workflows**: Search modes, metadata filters, annotations, and evaluation slices can be combined within the same workflow, making it possible to move from broad discovery to targeted curation without leaving the system. * **Extensibility**: New retrieval methods, annotation types, and evaluation protocols can be added incrementally within the same serving model. System Architecture ------------------- The server handles HTTP GET requests (pages and data) and POST requests (state mutations). Static HTML pages are served directly from disk. Data endpoints return JSON, which the browser renders client-side. The browser, the natural-language agent, and any programmatic client all use the same HTTP endpoints. It exposes a single server-mediated interface to multiple clients. The browser is the primary client, providing interactive access to search, annotation, curation, and evaluation workflows. It loads static frontend assets from the server and communicates with backend endpoints for all data access and state-changing operations. A WebSocket connection running alongside the HTTP server is used to push real-time notifications to the browser, for example broadcasting the results of long-running jobs as they complete. The same HTTP interface is also used by the natural-language agent and other programmatic clients. This keeps interactive and automated workflows aligned: agent-generated searches, browser-based exploration, and other external clients all operate on the same underlying abstractions and server logic. The natural-language agent maps plain-English queries into composed Wheel search requests, optionally using an LLM to rewrite or expand the query before submission. See :doc:`agent`. Endpoints fall into four groups: static pages, JSON data (search results, clip metadata, metrics, annotation state, plus Range-served video), CSV exports, and the POST actions that change state, such as annotation writes, label management, auto-labelling, classifier training, and clustering. Specialized Data Stores ^^^^^^^^^^^^^^^^^^^^^^^ Wheel uses a separate store for each modality rather than a single monolithic database. Each store is optimized for its specific retrieval or storage task. .. list-table:: :header-rows: 1 :widths: 22 18 18 42 * - Store - Modality - Technology - Purpose * - Annotation store - Annotations and metadata - SQLite - Per-clip labels (manual and auto-generated) and optional time ranges * - Caption store - Text captions - SQLite with full-text search - Per-clip captions with BM25 keyword retrieval * - Cosmos embeddings store - Video embeddings - FAISS - Text-to-video and video-to-video semantic retrieval * - Caption embeddings store - Caption embeddings - FAISS - Semantic retrieval over caption text * - Visual embeddings store - Frame embeddings (Florence-2 / SigLIP2) - FAISS - Frame-level visual similarity search * - Trajectory store - Ego trajectories - FAISS and NumPy - Shape-based and predicate-based trajectory retrieval * - Perception store - Object detections - In-memory NumPy - Per-clip object counts, distances, and angular positions * - Leaderboard store - Evaluation metrics - In-memory NumPy - Per-model, per-clip metric scores for leaderboard queries Search Modes ^^^^^^^^^^^^ Every mode is backed by one of the stores above and resolves to a set of clip IDs, which is what makes them freely composable. Recent results are cached in memory. See :doc:`search/index` for the modes themselves. Evaluation ^^^^^^^^^^ Evaluation happens entirely offline. Models are run externally and their per-model prediction files are written into the predictions directory, which the server loads at startup. There is no upload endpoint, so adding a model means placing its files on disk and restarting. See :doc:`evaluation/index` for the file layout. Extending Wheel --------------- Adding a new search modality follows a consistent pattern: 1. **Create a data store.** Implement a store class that loads or builds the index for the new modality (for example a FAISS index, a NumPy array, or a SQLite table) and exposes a retrieval method that returns a ranked or filtered set of clip IDs. 2. **Write a preprocessing script.** Build the store from raw data offline, following the same pattern as existing scripts under ``scripts/`` (e.g. ``extract_video_text_embeddings.py`` for embedding indexes, ``extract_trajectory_stats.py`` for the trajectory store). The server loads the prepared store at startup. 3. **Wire it into the search pipeline.** Register the new store in the server and add its retrieval logic to the search method so that it participates in the shared ranking and filter composition that all other modes use. Because every mode resolves to clip IDs and shares the same result structure, a new modality composes with existing search, annotation, and evaluation workflows with no further changes. Operational Considerations -------------------------- Preparation is separated from serving. Preprocessing, feature extraction, and index construction all happen ahead of time, so the server only has to read prepared indices. That is what keeps interactive latency low at tens of millions of clips. The server is configured via a YAML file passed on the command line. See :doc:`deployment` for the full configuration reference.