Stereolabs HDR, MAX, Neural Depth and TERRA Vision

There's a lot coming for outdoor robotic applications. Outdoor vision applications suffer from 2 major hurdles compared to indoor applications: variations in environmental light and exposure to the elements.

We can tell you from experience that very few - if not none - of 3D camera providers have been working on these two topics at the same time. Stereolabs decided to go all-in and created a new outdoor enclosure "PRO Rugged". We're seeing this being used mainly on outdoor vehicles - where the camera is exposed to the elements. It comes with an IP69K rating, a vibration resistant, tilting mount (we hope for teethed tilt adjustment), Gorilla glass and a sunshade. We'll have to wait for Q2'2025 for deliveries.

On the second front, both the ZED X One (the mono camera) and the ZED X (the stereo camera) got an HDR option, which makes use of the Sony ISX031 Automotive-grade HDR sensor. Now there is a catch, this is a Rolling Shutter camera and many of the existing cameras 'on wheels' - including the prior ZED X cameras - are Global Shutter cameras. The latter have the  advantage that every pixel has been recorded at the same time, and this eliminates 'tearing' when looking at (or being in) motion. But there is no such thing on the market as a 'Global Shutter HDR' chip. In this case, it's key that the time difference between reading the first row and the last row is as short as possible.

In addition, HDR chips are known to achieve the 'high dynamic range' by taking multiple images and combining these into one. But that leads to a second unwanted effect: blurring. The Sony ISX031 has a split-pixel technology that does away with this effect (as the picture shows).

We yet (at time of this writing) have to test how far we can take the HDR version on moving vehicles. Will both shutters on the ZED X Stereo be hardware synchronized, which is required for low-noise depth estimation? Will the distortion be manageable?

Much of the available Open Source robotics software for cameras - think about calibration, VIO and SLAM - work only well with global shutter cameras. It remains to be seen how well the HDR will perform in these applications.

It's not that a compact stereo camera can't see very far - it can -, it's that the distance measurement becomes extremely noisy, to the point where it becomes unusable. The baseline of the camera is largely to blame here (given good lens quality and sensor resolution), so Stereolabs decided to release a 'ZED X MAX' version of their cameras, adding roughly 50% more range due to a 51mm larger baseline (from 119mm to 170mm). We can understand demand for this type of sensor for the larger vehicles such as self-driving tractors or other off-road vehicles. A 23m range is still not that much if you compare it to LiDAR, which can commonly surpas 100m range - but if you'd like to get rid of your 25m range LiDAR, you come pretty close.

But! There is an even larger range solution possible. Stereolabs has created a tutorial explaining how two of these ZED X One cameras can be configured as a stereo-pair, effectively allowing for baselines allowing for very large ranges (think ~100m). 

There has been this thing about ZED X cameras that they only use 2 of the 4 GMSL2 cables. So if you take a look at the splitter in this picture, 2 connectors go to 2 ZED X cameras, and the other 2 are 'lying around' next to your box - they were unusable. This changed thanks to an update to the ZED Drivers. Version 1.3 supports all cables in use - if you operate the stereo cameras at half the speeds (15 FPS instead of 30 FPS) - OR your 2 extra cameras are only mono cameras (like the ZED X One), in that case the stereo cameras work at 30FPS and the mono cameras at 60FPS.