Agent Interface

Agent Interface#

The SIL-Wheel Agent exposes all of Wheel’s search and curation capabilities through a natural-language conversational interface. Users describe what they are looking for in plain English; the agent translates this into the appropriate combination of search modes and executes them.

The agent ships in the agent/ folder of the SIL-Wheel repository and can be connected to any running Wheel instance. Rather than navigating filter panels manually, users interact through natural language.

Example prompt:

Find clips with high curvature in rainy conditions where a pedestrian
is within 10 meters of the ego vehicle.

The agent decomposes this into trajectory, caption, and perception-based filters and executes them as a composed Wheel query.

Getting Started#

The agent ships no server and no credentials: point it at the Wheel instance you run or have access to by setting WHEEL_SERVER_URL (plus your WHEEL_USERNAME / WHEEL_PASSWORD for that server) in .env.

Option 1: via skill file (no repository clone required):

In any Cursor or Claude Code session, instruct the agent:

Set up the SIL Wheel Agent by following the skill file at
<SKILL-ENDPOINT>/skill.md (curl it, not fetch) and follow the install
instructions.

Replace <SKILL-ENDPOINT> with the location serving the agent skill for your deployment — for the public release that is the agent/SKILL.md file in the GitHub repository (or any HTTP endpoint you host it at for your own users).

Option 2: full clone:

git clone https://github.com/nv-tlabs/sil-wheel
cd sil-wheel/agent
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.template .env
# Edit .env: set WHEEL_SERVER_URL to your Wheel server, plus your
# WHEEL_USERNAME / WHEEL_PASSWORD for it

Verify the connection:

python examples/quickstart.py "construction zone in rain"

Then open the agent/ folder in Cursor or Claude Code and start chatting:

"Find me clips with hard braking in rainy conditions"

How It Works#

The agent is not a hosted service — it is an LLM running inside Cursor or Claude Code, guided by project rules and a Python client library (WheelClient). When a user describes a scenario in natural language, the LLM selects and composes the appropriate search modes from the full set available in Wheel, calls WheelClient methods, and returns results with clickable browser URLs.

For automated multi-strategy search, the built-in find_clips_for_scenario() method runs caption search, optional Cosmos semantic search, and matching classifier filters in parallel, then merges results via reciprocal rank fusion.

Search Modes#

The agent can compose any combination of Wheel’s 12 search modes in a single query. All modes are applied as an intersection (narrowing) pipeline.

Text-based retrieval

  • Caption search — full-text search over Qwen2.5-7B generated captions, with optional LLM query rewriting to expand coverage.

  • Cosmos embedding similarity — text-to-video and clip-to-clip retrieval using 768-dimensional Cosmos-Embed1 embeddings.

  • Visual similarity — frame-level text-to-image and image-to-video retrieval using Florence-2 / SigLIP2 embeddings.

Spatial and motion

  • Trajectory shape similarity — find kinematically similar driving patterns using full, 10-second, or 5-second comparison windows.

  • Trajectory predicates — filter by expressions on speed, curvature, acceleration, jerk (e.g., max(abs(curvature)) > 0.05).

  • Perception-based objects — filter by detected object class, count, distance, and angle relative to the ego vehicle.

Learned features

  • Classifier scores — filter using 93+ trained scenario classifiers (snow, construction zone, lane change, etc.) with configurable probability thresholds.

  • Cluster membership — filter by K-means cluster ID over Cosmos embeddings.

Metadata and annotations

  • Annotation labels — filter by 687+ manual and auto-generated labels with AND/OR logic, label exclusion, and annotation time filtering.

  • Numeric metric filtering — filter by model evaluation metric values.

  • Country / driving side — geographic filtering.

  • Data source — filter by dataset (MADS, MADS-1M, etc.).

Key Workflows#

Idea to clip IDs (for training teams)

The agent takes a natural-language scenario description and finds matching clips across multiple search strategies, returning ranked, deduplicated clip IDs ready for training pipeline integration:

client.find_clips_for_scenario("construction zone in rain", data_source="MADS")

Seed to expanded dataset

Starting from known good clips, the agent finds similar clips across visual and trajectory dimensions and iteratively expands a seed set:

client.expand_clip_set(["clip1", "clip2"], max_total=500)

Hard-example mining

The agent combines model evaluation feedback (scores from the leaderboard) with Wheel’s similarity measures to surface the most informative clips for the next training run. Retrieved or curated slices can directly become benchmarks or leaderboard subsets.

Composed queries

Modes that no single retrieval approach would surface on its own can be chained. For example, combining a trajectory curvature predicate with a learned classifier and a semantic similarity query can surface rare driving scenarios that are hard to describe in any single query.

CLI Usage#

The agent also provides a command-line interface for scripting:

# Caption search
python sil_wheel_agent/wheel_client.py search --caption "construction zone" -n 10

# Semantic text search
python sil_wheel_agent/wheel_client.py search --semantic-text "rainy highway at night" --data-source MADS

# Classifier filter
python sil_wheel_agent/wheel_client.py search --classifier "Snow" --threshold 0.7 -n 20

# Trajectory predicate
python sil_wheel_agent/wheel_client.py search --speed-expr "max(abs(curvature)) > 0.05" -n 10

# Find similar clips
python sil_wheel_agent/wheel_client.py similar dd87da72-... -n 20

# Multi-strategy scenario search
python sil_wheel_agent/wheel_client.py scenario "construction zone in rain" --data-source MADS

# Expand a seed set
python sil_wheel_agent/wheel_client.py expand --clips seeds.txt -o expanded.txt --max-total 500

# Export clip IDs
python sil_wheel_agent/wheel_client.py export --caption "tunnel" -o tunnel_clips.txt

# Model leaderboard
python sil_wheel_agent/wheel_client.py metrics

Python API#

from sil_wheel_agent import WheelClient

client = WheelClient()
client.login()

# Compose multiple search modes in one call
total, results = client.search(
    data_source="MADS",
    search="highway merge",
    query_rewrite=True,
    classifier_select="interesting",
    probability_threshold=0.5,
)

# Results include browser URLs
for r in results:
    print(f"{r.clip_id} score={r.best_score}")
    print(f"  → {client.clip_url(r.clip_id)}")

# Format as markdown table with links
print(client.format_results_with_urls(results))

The WheelClient supports parallel pagination, thread-safe caching (5-minute TTL for classifiers and stats), automatic retry with re-login, and both production and dev server connections.

Discovery Methods#

The client provides methods for exploring the available data:

  • get_data_sources() — available datasets and their sizes

  • get_classifiers() / list_classifier_names() — trained classifiers

  • get_metrics() / get_leaderboard() — model evaluation results

  • get_annotations_summary() — label vocabulary and counts

  • scenario_inventory() — overview of available scenarios per data source

  • lookup_clip() / lookup_clips_batch() — detailed info for specific clips

Server Policy#

  • Read-only mode — set WHEEL_READONLY=1 to make the agent refuse all write operations (annotation uploads, classifier training, auto-labeling, clustering, and label management). Use this for a shared or production server.

  • Writes — allowed by default (you own your server). They target whatever WHEEL_SERVER_URL (or WHEEL_DEV_URL) points at. Credentials live in .env and are never printed or logged.

In-browser Onboarding Assistant#

A separate, lighter-weight assistant ships inside the Wheel UI itself. Every Wheel page renders a “Ask SIL-Wheel” launcher in the lower right; opening it reveals a chat panel titled SIL-Wheel Onboarding Assistant that answers documentation-grounded questions about how to use Wheel. Unlike the search agent above, this assistant does not call Wheel APIs or run searches: it only reads the project documentation and returns explanations and pointers to the relevant doc pages.

The assistant is backed by the standalone sil-wheel-docs-agent server, configured through the server.agent_url key in the launch YAML (see Deployment). When agent_url is unset or the configured server is unreachable, the launcher hides itself rather than showing a broken control, so the chatbot only appears in deployments where it has been wired up.

Use the in-browser assistant for “how do I” questions while working in Wheel. Use the natural-language search agent above when you want to translate a scenario description into actual clip results.

Source#

The agent ships in the agent/ folder of the SIL-Wheel repository: github.com/nv-tlabs/sil-wheel. For questions or issues, open an issue on the GitHub repository.