ZED X and NVIDIA: Applications, Architecture and Integrations

Intel's decision to not support ARM/Jetson platforms with its RealSense cameras can be viewed as a strategic choice to prioritize their x86 architecture, reinforcing their market position in environments where their processors are prevalent. Stereolabs capitalized on this competitive delineation by ensuring that the ZED camera was ARM/Jetson compatible, providing a valuable alternative to those seeking interoperability outside of Intel’s ecosystem. This foresight by Stereolabs to complement NVIDIA's ARM architecture allowed them to carve out a unique space in the market, turning what could be seen as Intel’s competitive exclusion into a strategic opportunity for themselves.

Stereolabs' bet on ARM's expansive potential was serendipitously aligned with NVIDIA's evolution. NVIDIA had broadened its horizon from its GPU-centric legacy to revolutionize robotics with its Isaac SDK and Jetson hardware. Their dedication to integrating full-stack 2D/3D vision and prioritizing GPU-offloaded image processing functions created a distinct niche that Intel had yet to embrace.

With NVIDIA, Stereolabs found a kindred spirit in their pursuit of versatility and innovation. With the ZED X camera, Stereolabs went all-in with NVIDIA, no longer supporting this camera on Intel platforms. Oh irony!

The exclusive ZED X-NVIDIA combo has made a bet on a few high-profile applications in robotics. We have highlighted here the three most dominantly supported applications for the ZED X.

As intriguing as these applications are, you might be wondering, "How do I actually get started?" The next chapter will delve into the essential building blocks you'll need to bring these applications to life. From setting up the development environment to choosing the right SDKs and hardware, we'll guide you through the foundational steps to ensure a successful project.

Embarking on a project that leverages the ZED X camera and NVIDIA Jetson platforms requires a well-thought-out architecture. In this chapter, we'll outline the key components you'll need to set up your development environment and target hardware and software runtime.

We outlined in this diagram the software and components of a typical setup. You can develop code from any Operating System, using a remote connection to a target (any Jetson device). But you do need to setup this target device with the NVIDIA sdkmanager. Most developers nowadays run Docker instances on the Jetson target in order to keep the complex dependencies equal among all their setups.

Here’s the breakdown:

ZED SDK: The ZED SDK v4 is your primary tool for interacting with the ZED X camera. It provides APIs for capturing images, generating depth maps, and managing the camera devices.

ROS Humble: We recommend to stick to ROS 2 and the Humble release, as it is best supported on NVIDIA hardware and is a requirement if you want to add Isaac ROS later-on, since the Isaac SDK is not required.

Docker: You will run Docker containers both on your target hardware and development environment, even if these are already running the Ubuntu 20.04 Operating System.

NVIDIA SDK Manager: Only required to bootstrap your Jetson device (it will install the correct OS and libraries). Once the bootstrapping is done, you don’t need the SDK manager anymore and any developer can use the Docker containers instead, on any supported Docker OS.

The ZED X camera offers a variety of integration options that cater to different development needs. In this chapter, we'll delve into some of the most popular frameworks in the mobile robotics community: ZED ROS2, ZED PCL, ZED Docker, and ZED OpenCV.

Highlights:

• ZED ROS node: The wrapper provides access to left and right rectified/unrectified images, depth data, colored 3D point clouds, IMU data, VIO pose tracking, detected objects and human body skeleton.
• GNSS Fusion: The integration allows for GNSS data fusion, enabling more accurate robot localization in the presence of a ROS 2 driver for a GNSS sensor.
• 2D Mode: For robots operating on a planar surface, a 2D mode is available, and fixes the Z coordinate for odometry and pose to a constant value.
• Object and Body Tracking: The object detection pipeline offered by the ZED SDK is made available by the ZED ROS node.
• Dynamic Reconfigure: The ZED camera can be configured in rqt using the Dynamic Reconfigure Plugin, allowing you to inspect many of the parameters of the camera.

Limitations:

• Requires ZED SDK v4.0.6 or later
• No Docker container provided for Humble on 20.04.

The ZED ROS2 integration offers a comprehensive solution for those looking to leverage the capabilities of ZED cameras in a ROS2 environment.

Highlights:

• Seamless Conversion: The example program uses a conversion function to map between sl::Mat and cv::Mat, allowing for a seamless transition between ZED SDK and OpenCV.
• Flexible Configuration: The program allows for various configurations, including camera resolution, depth mode, and coordinate units, making it adaptable to different use-cases.
• CUDA Support: For those with CUDA-enabled devices, the program also includes a conversion function for cv::cuda::GpuMat, enabling GPU-accelerated image processing.

Limitations:

• None we could identify.

This ZED OpenCV integration offers a straightforward way to integrate ZED cameras with OpenCV, especially for those who prefer not to use ROS.

With this, we've covered the popular framework integrations for the ZED X camera. Each has its own set of capabilities and limitations, making it crucial to choose the one that aligns best with your project's specific needs.

Hopefully you became a bit wiser upon the ins and outs of the using the ZED X on an NVIDIA target. Intermodalics offers professional support on this and other integrations, don’t hesitate to reach out !