ZED X and NVIDIA: Applications, Architecture and Integrations

For every mobile robot, the integration of reliable sensors with an efficient computing platform is crucial for achieving modern-day capabilities. The ZED X stereo camera, released by Stereolabs in 2023 and specifically designed for compatibility with NVIDIA Jetson, offers a compelling solution for both indoor and outdoor robotic applications. Building on our previous explorations of NVIDIA Isaac ROS, this guide aims to provide a thorough understanding of how the ZED X camera can be effectively utilized in various scenarios.

The ZED X and X mini 3D camera

The ZED X and ZED X mini stereo camera lineup for mobile robotics applications.

A tale of Accessibility and Choice

In the young landscape of 3D vision technology, Stereolabs launched the ZED 3D camera in 2015, amidst the rise of Intel's RealSense 3D camera family. The early days saw ZED cameras compatible with Intel systems, a market Intel aimed to dominate with the launch of its third-generation RealSense D400 series in January 2018. However, Intel's RealSense cameras were not officially supported on ARM/Jetson platforms, a decision that revealed an opportunity for Stereolabs.

Two humanoid robots in a stereolabs zed versus intel realsense battle

Intel's decision to not support ARM/Jetson platforms with its RealSense cameras can be viewed as a strategic choice to prioritize their x86 architecture, reinforcing their market position in environments where their processors are prevalent. Stereolabs capitalized on this competitive delineation by ensuring that the ZED camera was ARM/Jetson compatible, providing a valuable alternative to those seeking interoperability outside of Intel’s ecosystem. This foresight by Stereolabs to complement NVIDIA's ARM architecture allowed them to carve out a unique space in the market, turning what could be seen as Intel’s competitive exclusion into a strategic opportunity for themselves.

Stereolabs' bet on ARM's expansive potential was serendipitously aligned with NVIDIA's evolution. NVIDIA had broadened its horizon from its GPU-centric legacy to revolutionize robotics with its Isaac SDK and Jetson hardware. Their dedication to integrating full-stack 2D/3D vision and prioritizing GPU-offloaded image processing functions created a distinct niche that Intel had yet to embrace.

With NVIDIA, Stereolabs found a kindred spirit in their pursuit of versatility and innovation. With the ZED X camera, Stereolabs went all-in with NVIDIA, no longer supporting this camera on Intel platforms. Oh irony!

Chosen Applications of ZED X and NVIDIA Jetson

The exclusive ZED X-NVIDIA combo has made a bet on a few high-profile applications in robotics. We have highlighted here the three most dominantly supported applications for the ZED X.

Indoor Navigation and Mapping

One of the principal features of the ZED X camera is its ability to produce depth maps, even in indoor lighting conditions. When coupled with NVIDIA Isaac ROS capabilities, this becomes a useful tool for indoor navigation. Popular applications are mapping out complex environments, navigation, avoiding obstacles, or recognizing specific objects or areas of interest. The Isaac ROS SDK is officially supporting the ZED camera lineup for their indoor applications.

a mobile robot navigating indoor in a factory with the zed camera

Outdoor Exploration and Construction

The rugged design of the ZED X camera, complete with its IP66-rated enclosure, makes it suitable for outdoor applications. For example, agricultural drones capturing real-time data on crop health or automated mining and construction robots. NVIDIA does not offer many tools in their Isaac SDK for outdoor applications, and you'll need to build on the ZED SDK for these outdoor applications, which allows for GNSS integration, object detection and recognition (more on that below) and 3D mesh building.

a forrest road that is 3d meshed with the ZED camera

Object Detection and Recognition

The ZED X camera comes with an SDK with a 3D object detection pipeline. It uses the NVIDIA Jetson GPU to run neural networks for real-time object detection and recognition. It offers built-in detectors for persons, vehicles, bags, animals, electronics or fruit. If that does not cut your needs, it is possible to create a Custom Detector using the NVIDIA TensorRT library.

A person running on a walkway who is being detected with the zed camera AI model

As intriguing as these applications are, you might be wondering, "How do I actually get started?" The next chapter will delve into the essential building blocks you'll need to bring these applications to life. From setting up the development environment to choosing the right SDKs and hardware, we'll guide you through the foundational steps to ensure a successful project.

Architecture - Setting Up Your Development Environment

Embarking on a project that leverages the ZED X camera and NVIDIA Jetson platforms requires a well-thought-out architecture. In this chapter, we'll outline the key components you'll need to set up your development environment and target hardware and software runtime.

Essential SDKs and Components

We outlined in this diagram the software and components of a typical setup. You can develop code from any Operating System, using a remote connection to a target (any Jetson device). But you do need to setup this target device with the NVIDIA sdkmanager. Most developers nowadays run Docker instances on the Jetson target in order to keep the complex dependencies equal among all their setups.

A schematic of the NVIDIA ZED X Software Architecture with on-Target development set up

A Typical ZED X + NVIDIA setup.

Here’s the breakdown:

ZED SDK: The ZED SDK v4 is your primary tool for interacting with the ZED X camera. It provides APIs for capturing images, generating depth maps, and managing the camera devices.

ROS Humble: We recommend to stick to ROS 2 and the Humble release, as it is best supported on NVIDIA hardware and is a requirement if you want to add Isaac ROS later-on, since the Isaac SDK is not required.

Docker: You will run Docker containers both on your target hardware and development environment, even if these are already running the Ubuntu 20.04 Operating System.

NVIDIA SDK Manager: Only required to bootstrap your Jetson device (it will install the correct OS and libraries). Once the bootstrapping is done, you don’t need the SDK manager anymore and any developer can use the Docker containers instead, on any supported Docker OS.

Target Hardware - Best choice

We recommend to get started with the NVIDIA Jetson AGX Orin 64GB Developer Kit, with the mandatory GMSL2 Capture Card, which can only run 2 ZED X devices (since each ZED devices has 2 cameras, the maximum is 4 cameras). It offers you 64GB RAM on 12 CPU Cores, 275 TOPS of AI Performance and a 2048 Cuda cores. The advantage is that it can also serve as your build environment, since it’s sufficiently powerful to compile applications and it avoids setting up a cross compiler environment on your regular Linux workstation.

A picture of the Jetson AGX Orin developer kit

The NVIDIA Jetson AGX Orin Developer Kit

Once you are familiar and start running first applications, you can trial field test on the ZED Box Orin NX 16GB, which also includes a GMSL2 port. It is a bit cheaper and lighter than the Developer kit and offers 16GB of RAM, on an 8 core CPU, 100TOPS AI performance, 1024 Cuda cores and 32 Tensor cores. Since NVIDIA is taking care of the bootstrapping, you can also opt for (cheaper) non-ZED field devices.

A picture of the ZED Box Orin NX device

The ZED Box Orin NX 16GB

Framework Integrations for ZED X

The ZED X camera offers a variety of integration options that cater to different development needs. In this chapter, we'll delve into some of the most popular frameworks in the mobile robotics community: ZED ROS2, ZED PCL, ZED Docker, and ZED OpenCV.

ZED ROS2 Integration

The ZED ROS2 integration allows you to use ZED stereo cameras with ROS2, providing access to a wide range of data including rectified/unrectified images, depth data, colored 3D point clouds, position and mapping with GNSS data fusion. Since we recommend using ROS 2 Humble, care must be taken to not use the Docker containers of the ZED ROS 2 wrapper but build instead on top of ROS Humble in an Ubuntu 20.04 Docker container.

The ROS RVIZ tool showing a ZED point cloud


  • ZED ROS node: The wrapper provides access to left and right rectified/unrectified images, depth data, colored 3D point clouds, IMU data, VIO pose tracking, detected objects and human body skeleton.
  • GNSS Fusion: The integration allows for GNSS data fusion, enabling more accurate robot localization in the presence of a ROS 2 driver for a GNSS sensor.
  • 2D Mode: For robots operating on a planar surface, a 2D mode is available, and fixes the Z coordinate for odometry and pose to a constant value.
  • Object and Body Tracking: The object detection pipeline offered by the ZED SDK is made available by the ZED ROS node.
  • Dynamic Reconfigure: The ZED camera can be configured in rqt using the Dynamic Reconfigure Plugin, allowing you to inspect many of the parameters of the camera.


  • Requires ZED SDK v4.0.6 or later
  • No Docker container provided for Humble on 20.04.

The ZED ROS2 integration offers a comprehensive solution for those looking to leverage the capabilities of ZED cameras in a ROS2 environment.

ZED PCL Integration

The PCL 3D viewer showing a Point Cloud with keypoints

The ZED PCL integration serves as an example program that demonstrates how to acquire and display a 3D point cloud using the Point Cloud Library (PCL). Unlike many PCL applications that rely on ROS for data acquisition and processing, this example specifically shows how to achieve this without using ROS.


  • Standalone Operation: This example operates independently of ROS, making it a unique resource for developers who want to use PCL without the ROS ecosystem.
  • Broad OS support: The prerequisites for running this example are Ubuntu 16.04, ZED SDK 3, and CUDA. The PCL library with its visualization module is also required.
  • Live view or SVO playback: This example shows how to load camera data from a pre-recorded SVO file or how to read data from a live camera.


  • CPU based Conversion: The conversion of the point cloud to PCL format can be time-consuming, since it is done on the CPU, which may affect the application's running time.

This ZED PCL integration offers an easy approach to 3D point cloud acquisition and display, especially for those who prefer not to use ROS.

ZED Docker Integration

The ZED Docker integration provides a virtualized environment for running the ZED SDK, isolating it from the host machine. This is particularly useful for developers who need to manage lots of dependencies and want to start from a tried-and-tested target environment.

For the NVIDIA Jetson workflow, both the Development and the Target run Docker containers, even if the host OS is a supported Ubuntu version. The dependencies are too intricate and Docker guarantees a reproducible development and runtime environment. Both NVIDIA and Stereolabs offer Docker images in order to offer stable environments.


  • Jetson Development and Runtime: The Docker images allow to setup a development environment on the Jetson target (for example, on an AGX Orin target), which removes the complexity of cross compiling or emulating for the ARM target.
  • ARM Emulation on x86: For those wanting to build ARM software on an x86_64 platform, the NVIDIA Jetson Docker image, along with QEMU, allows you to build the application on a fast x86 host and launch it on a resource contrained Jetson device.


  • Graphical User Interfaces: The Docker images will open access to the NVIDIA hardware, but will not setup the environment for a graphical user interface
  • CUDA-Accelerated Applications: While it's possible to run aarch64 programs on an x86_64 workstation using QEMU, building (and running) CUDA-accelerated applications require additional steps and a Linux for Tegra (L4T) base image.

ZED OpenCV Integration

The ZED OpenCV integration serves as an example program that demonstrates how to capture images and depth maps from the ZED camera and convert them into OpenCV-compatible formats. This is particularly useful for developers who are already familiar with OpenCV and want to leverage its capabilities for image and depth map processing.


  • Seamless Conversion: The example program uses a conversion function to map between sl::Mat and cv::Mat, allowing for a seamless transition between ZED SDK and OpenCV.
  • Flexible Configuration: The program allows for various configurations, including camera resolution, depth mode, and coordinate units, making it adaptable to different use-cases.
  • CUDA Support: For those with CUDA-enabled devices, the program also includes a conversion function for cv::cuda::GpuMat, enabling GPU-accelerated image processing.


  • None we could identify.

This ZED OpenCV integration offers a straightforward way to integrate ZED cameras with OpenCV, especially for those who prefer not to use ROS.

With this, we've covered the popular framework integrations for the ZED X camera. Each has its own set of capabilities and limitations, making it crucial to choose the one that aligns best with your project's specific needs.

Next steps?

Hopefully you became a bit wiser upon the ins and outs of the using the ZED X on an NVIDIA target. Intermodalics offers professional support on this and other integrations, don’t hesitate to reach out !

Order a Robot Vision capability

Discover Our Expertise

Camera Calibration

Extrinsic and Intrinsic Camera Calibration. We use the right camera model, calibration pattern and algorithms for your specific use case.

Read Camera Calibration

Visual-Inertial Odometry (VIO)

Our VIO outperforms what is available in common libraries and hardware implementations. We implement monocular, stereo and multi-camera VIO solutions

Read Visual-Inertial Odometry (VIO)

2D/3D Collision Avoidance

We create global and local planners that take into account the collision of the robot body with the environment.

Read 2D/3D Collision Avoidance