.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. .. SPDX-License-Identifier: Apache-2.0 .. .. Licensed under the Apache License, Version 2.0 (the "License"); .. you may not use this file except in compliance with the License. .. You may obtain a copy of the License at .. .. http://www.apache.org/licenses/LICENSE-2.0 .. .. Unless required by applicable law or agreed to in writing, software .. distributed under the License is distributed on an "AS IS" BASIS, .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. .. See the License for the specific language governing permissions and .. limitations under the License. Agent Interface =============== The SIL-Wheel Agent exposes all of Wheel's search and curation capabilities through a natural-language conversational interface. Users describe what they are looking for in plain English; the agent translates this into the appropriate combination of search modes and executes them. The agent ships in the ``agent/`` folder of the SIL-Wheel repository and can be connected to any running Wheel instance. Rather than navigating filter panels manually, users interact through natural language. Example prompt:: Find clips with high curvature in rainy conditions where a pedestrian is within 10 meters of the ego vehicle. The agent decomposes this into trajectory, caption, and perception-based filters and executes them as a composed Wheel query. Getting Started --------------- The agent ships no server and no credentials: point it at the Wheel instance you run or have access to by setting ``WHEEL_SERVER_URL`` (plus your ``WHEEL_USERNAME`` / ``WHEEL_PASSWORD`` for that server) in ``.env``. **Option 1: via skill file** (no repository clone required): In any Cursor or Claude Code session, instruct the agent:: Set up the SIL Wheel Agent by following the skill file at /skill.md (curl it, not fetch) and follow the install instructions. Replace ```` with the location serving the agent skill for your deployment — for the public release that is the ``agent/SKILL.md`` file in the GitHub repository (or any HTTP endpoint you host it at for your own users). **Option 2: full clone**:: git clone https://github.com/nv-tlabs/sil-wheel cd sil-wheel/agent python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt cp .env.template .env # Edit .env: set WHEEL_SERVER_URL to your Wheel server, plus your # WHEEL_USERNAME / WHEEL_PASSWORD for it Verify the connection:: python examples/quickstart.py "construction zone in rain" Then open the ``agent/`` folder in Cursor or Claude Code and start chatting:: "Find me clips with hard braking in rainy conditions" How It Works ------------ The agent is not a hosted service — it is an LLM running inside Cursor or Claude Code, guided by project rules and a Python client library (``WheelClient``). When a user describes a scenario in natural language, the LLM selects and composes the appropriate search modes from the full set available in Wheel, calls ``WheelClient`` methods, and returns results with clickable browser URLs. For automated multi-strategy search, the built-in ``find_clips_for_scenario()`` method runs caption search, optional Cosmos semantic search, and matching classifier filters in parallel, then merges results via reciprocal rank fusion. Search Modes ------------ The agent can compose any combination of Wheel's 12 search modes in a single query. All modes are applied as an intersection (narrowing) pipeline. **Text-based retrieval** - **Caption search** — full-text search over Qwen2.5-7B generated captions, with optional LLM query rewriting to expand coverage. - **Cosmos embedding similarity** — text-to-video and clip-to-clip retrieval using 768-dimensional Cosmos-Embed1 embeddings. - **Visual similarity** — frame-level text-to-image and image-to-video retrieval using Florence-2 / SigLIP2 embeddings. **Spatial and motion** - **Trajectory shape similarity** — find kinematically similar driving patterns using full, 10-second, or 5-second comparison windows. - **Trajectory predicates** — filter by expressions on speed, curvature, acceleration, jerk (e.g., ``max(abs(curvature)) > 0.05``). - **Perception-based objects** — filter by detected object class, count, distance, and angle relative to the ego vehicle. **Learned features** - **Classifier scores** — filter using 93+ trained scenario classifiers (snow, construction zone, lane change, etc.) with configurable probability thresholds. - **Cluster membership** — filter by K-means cluster ID over Cosmos embeddings. **Metadata and annotations** - **Annotation labels** — filter by 687+ manual and auto-generated labels with AND/OR logic, label exclusion, and annotation time filtering. - **Numeric metric filtering** — filter by model evaluation metric values. - **Country / driving side** — geographic filtering. - **Data source** — filter by dataset (MADS, MADS-1M, etc.). Key Workflows ------------- **Idea to clip IDs** (for training teams) The agent takes a natural-language scenario description and finds matching clips across multiple search strategies, returning ranked, deduplicated clip IDs ready for training pipeline integration: .. code-block:: python client.find_clips_for_scenario("construction zone in rain", data_source="MADS") **Seed to expanded dataset** Starting from known good clips, the agent finds similar clips across visual and trajectory dimensions and iteratively expands a seed set: .. code-block:: python client.expand_clip_set(["clip1", "clip2"], max_total=500) **Hard-example mining** The agent combines model evaluation feedback (scores from the leaderboard) with Wheel's similarity measures to surface the most informative clips for the next training run. Retrieved or curated slices can directly become benchmarks or leaderboard subsets. **Composed queries** Modes that no single retrieval approach would surface on its own can be chained. For example, combining a trajectory curvature predicate with a learned classifier and a semantic similarity query can surface rare driving scenarios that are hard to describe in any single query. CLI Usage --------- The agent also provides a command-line interface for scripting: .. code-block:: bash # Caption search python sil_wheel_agent/wheel_client.py search --caption "construction zone" -n 10 # Semantic text search python sil_wheel_agent/wheel_client.py search --semantic-text "rainy highway at night" --data-source MADS # Classifier filter python sil_wheel_agent/wheel_client.py search --classifier "Snow" --threshold 0.7 -n 20 # Trajectory predicate python sil_wheel_agent/wheel_client.py search --speed-expr "max(abs(curvature)) > 0.05" -n 10 # Find similar clips python sil_wheel_agent/wheel_client.py similar dd87da72-... -n 20 # Multi-strategy scenario search python sil_wheel_agent/wheel_client.py scenario "construction zone in rain" --data-source MADS # Expand a seed set python sil_wheel_agent/wheel_client.py expand --clips seeds.txt -o expanded.txt --max-total 500 # Export clip IDs python sil_wheel_agent/wheel_client.py export --caption "tunnel" -o tunnel_clips.txt # Model leaderboard python sil_wheel_agent/wheel_client.py metrics Python API ---------- .. code-block:: python from sil_wheel_agent import WheelClient client = WheelClient() client.login() # Compose multiple search modes in one call total, results = client.search( data_source="MADS", search="highway merge", query_rewrite=True, classifier_select="interesting", probability_threshold=0.5, ) # Results include browser URLs for r in results: print(f"{r.clip_id} score={r.best_score}") print(f" → {client.clip_url(r.clip_id)}") # Format as markdown table with links print(client.format_results_with_urls(results)) The ``WheelClient`` supports parallel pagination, thread-safe caching (5-minute TTL for classifiers and stats), automatic retry with re-login, and both production and dev server connections. Discovery Methods ^^^^^^^^^^^^^^^^^ The client provides methods for exploring the available data: - ``get_data_sources()`` — available datasets and their sizes - ``get_classifiers()`` / ``list_classifier_names()`` — trained classifiers - ``get_metrics()`` / ``get_leaderboard()`` — model evaluation results - ``get_annotations_summary()`` — label vocabulary and counts - ``scenario_inventory()`` — overview of available scenarios per data source - ``lookup_clip()`` / ``lookup_clips_batch()`` — detailed info for specific clips Server Policy ------------- - **Read-only mode** — set ``WHEEL_READONLY=1`` to make the agent refuse all write operations (annotation uploads, classifier training, auto-labeling, clustering, and label management). Use this for a shared or production server. - **Writes** — allowed by default (you own your server). They target whatever ``WHEEL_SERVER_URL`` (or ``WHEEL_DEV_URL``) points at. Credentials live in ``.env`` and are never printed or logged. In-browser Onboarding Assistant ------------------------------- A separate, lighter-weight assistant ships inside the Wheel UI itself. Every Wheel page renders a "Ask SIL-Wheel" launcher in the lower right; opening it reveals a chat panel titled **SIL-Wheel Onboarding Assistant** that answers documentation-grounded questions about how to use Wheel. Unlike the search agent above, this assistant does not call Wheel APIs or run searches: it only reads the project documentation and returns explanations and pointers to the relevant doc pages. The assistant is backed by the standalone ``sil-wheel-docs-agent`` server, configured through the ``server.agent_url`` key in the launch YAML (see :doc:`deployment`). When ``agent_url`` is unset or the configured server is unreachable, the launcher hides itself rather than showing a broken control, so the chatbot only appears in deployments where it has been wired up. Use the in-browser assistant for "how do I" questions while working in Wheel. Use the natural-language search agent above when you want to translate a scenario description into actual clip results. Source ------ The agent ships in the ``agent/`` folder of the SIL-Wheel repository: `github.com/nv-tlabs/sil-wheel `_. For questions or issues, open an issue on the GitHub repository.