[Interactive Drive] Add Vulkan Support

[ArielG-NV]

The new Vulkan backend (via very non-rigorous testing) is significantly faster than CUDA (hits lows of 2ms, highs of ~10ms; CUDA has lows of around 10ms)

The design was rushed on my end, so I do not have a proper PR description to provide accompanying the code added.

At a high-level the goal was: minimize syncs (versus last-time); minimize use external memory for a 'zero copy' abstraction over CUDA; do not recreate buffers every time we enter vk-ludus. It seems these design changes had massive perf-benefits when put together.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

The PR adds an optional Vulkan renderer and integrates it into Interactive Drive, including CUDA/Vulkan external-memory synchronization and reusable output and staging pools. The prior per-slot dimension and output-consumer synchronization issues are addressed, but staging retrieval failures can still exhaust the staging pool.

• Adds Vulkan task/mesh/fragment shaders and a compute export path.
• Adds CUDA-visible Vulkan buffers, timeline-semaphore ordering, and Python bindings.
• Adds Vulkan backend selection to Interactive Drive configuration and launch manifests.
• Adds per-slot output ownership and staging metadata management.

Confidence Score: 4/5

The PR is not yet safe to merge because exceptions during staging retrieval or JPEG encoding can permanently exhaust the staging pool and stop staged rendering.

Staging slots are marked occupied before rendering, but both consumer paths release them only after potentially throwing operations; two caught failures leave no slot available for subsequent frames.

Files Needing Attention: integrations/omnidreams/ludus-renderer/ludus_renderer/_cpp/bindings/torch_rasterize_vk.cpp

Important Files Changed

Filename	Overview
integrations/omnidreams/ludus-renderer/ludus_renderer/_cpp/bindings/torch_rasterize_vk.cpp	Adds the public Vulkan binding and output/staging lease management; staging consumer exceptions can permanently leak both staging slots.
integrations/omnidreams/ludus-renderer/ludus_renderer/_cpp/render/ludus_timestamped_vk.cpp	Implements Vulkan rendering, per-slot staging metadata, external-memory exports, and CUDA/Vulkan synchronization.
integrations/omnidreams/ludus-renderer/ludus_renderer/_cpp/common/vkutil.cpp	Implements Vulkan device selection, external-memory/semaphore interop, and resource lifecycle helpers.
integrations/omnidreams/ludus-renderer/ludus_renderer/_ops/context_vk.py	Adds the Python Vulkan timestamped context and packs renderer inputs into the native layouts.
integrations/omnidreams/omnidreams/interactive_drive/rasterizer.py	Selects the CUDA or Vulkan Ludus context according to the configured backend.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  R[render_to_staging] --> A[Acquire one of two staging slots]
  A --> V[Mark stagingValid]
  V --> C[Render and copy frame]
  C --> G[get_staging_data or JPEG encode]
  G -->|success| F[Release staging slot]
  G -->|exception| L[Slot remains valid]
  L --> E[Two failures exhaust pool]
  E --> N[Later render_to_staging returns false]

Loading

_{Reviews (8): Last reviewed commit: "more robustness" | Re-trigger Greptile}

[ArielG-NV]

Note it looks like there is a race-condition in the code (rotating buffers without checking 'frame-inflight'), this needs to be solved before code is RFR

[ArielG-NV]

/ok to test ce5b00a

[ArielG-NV]

/ok to test 6602176

[ArielG-NV]

/ok to test 7a72bf0

[ArielG-NV]

/ok to test 713f5ac

[greptile-apps]

Staging failures leak slots

When staging retrieval or JPEG encoding throws during synchronization, copying, allocation, or encoding, releaseStagingSlot is skipped and the slot remains occupied. After two such failures, subsequent staged renders return (-1, false) because both slots remain unavailable.