Stereo Vision

Bottlenose is a full 4K RGB-D camera. The stereo version of Bottlenose can provide disparity and color images (RGB-D) as well as a triangulated sparse point cloud from matching keypoints. In order for this process to produce accurate results, Bottlenose has to undergo a calibration process. There are many software packages available that can take care of the calibration once the required image data is collected. In this guide we used the calib tool from calib.io.

Calibration Data Acquisition

Data acquisition is the first step of the calibration process. Bottlenose cameras can be configured to stream images at different resolutions ranging from VGA to 4K.

👍

Please calibrate once a desired resolution is selected. You will have to recalibrate when resolution, scaling, cropping, and other image related settings are changed.

We recommend calibrating your Bottlenose after selecting your desired resolution, scaling, and cropping settings.

It is important to note that some calibration suites (e.g. calib.io) may be very sensitive to image quality. Therefore it is a good practice to ensure that the camera produces well-focused and clear images for the calibration process.

Bottlenose is a GigE Vision compliant camera. This means any GigE Vision compliant tool such as the Pleora eBusPlayer can be used to receive and save images from Bottlenose. The Pleora eBusPlayer is a graphical tool that can be used to control the Bottlenose camera. It provides full access to Bottlenose registers with the ability to stream and save images from the camera.

Since eBusPlayer cannot directly handle stereo images, Bottlenose also comes with a tool to visualize and save stereo images for calibration.

Data Acquisition with Bottlenose Calibrator

If you are using the Bottlenose Calibrator to acquire calibration data, follow these steps. This assumes the tool is properly installed on your system already.

1. Launch the Bottlenose Calibrator utility
2. Click on Connect to connect to your Bottlenose camera.
3. Make sure Acquire Calibration Data is selected
4. Click on the Start button to start acquisition
5. Set the data folder to your desired location
6. Click the Save button for every pair of images you would like to save.

Images acquired for calibration must include a calibration target. Calibration targets come in various sizes and shapes. The choice of target generally depends on your application and expected accuracy. In this guide, we used a basic checkerboard target for calibration. The size of each cell on the board should be known. In the example provided below, each cell is a 50mm x 50mm square. It is good practice to have the target cover the field of view at different locations and depths. Images can be acquired from Bottlenose using its GigE Vision interface or API. The Bottlenose acquisition tool automatically creates two folders to keep images from the left and right sensors separate.

Calibration Process

The calibration process consists of feeding the calibration images into a calibration tool such as calib by calib.io to generate intrinsic and extrinsic parameters for the camera.

Pinhole Model

Bottlenose supports the pinhole camera model taking into account the radial and tangential distortion factors. Radial distortion causes straight lines to appear curved. Radial distortion becomes larger the farther points are from the center of the image. Radial distortion can be represented as follows:

Similarly, tangential distortion occurs because the image-taking lense is not aligned perfectly parallel to the imaging plane. So, some areas in the image may look nearer than expected. The amount of tangential distortion can be represented as below:

In order to remove distortion from images, Bottlenose requires the following five parameters that constitute the distortion coefficients:

Intrinsics and Extrinsics

Once calibrated, the intrinsic and extrinsic parameters along with the distortion coefficients can be saved back into the camera. The intrinsic parameters include the focal length and the optical center of the camera and help in removing lens distortion. The extrinsic parameters contain the rotation and the translation vectors of the camera and help translating the coordinate of a 3D point to a coordinate system. Bottlenose uses these parameters to internally compute its undistortion, rectification, and 3D reprojection parameters.

After calibration Bottlenose is able to produce undistorted and stereo-rectified images. For external processing an end-user may require the Bottlenose camera to transmit back the internally generated undistortion, rectification, and reprojection data.

Calibration Parameters

Once calibrated, the generated calibration parameters can now be uploaded onto the Bottlenose camera for computation. The calibration parameters for each individual sensor inside a Bottlenose consist of the following values:

• Horizontal focal length, fx:
• Vertical focal length, fy: in general, fy is equal to fx.
• Horizontal optical center, cx
• Vertical optical center, cy
• Radial distortion parameters, k1, k2, and k3
• Tangential distortion parameters, p1 and p2
• Rotation vector, rx, ry, and rz
• Translation vector, tx, ty, and tz
• Calibrated image width and height.

When the calibration parameters are loaded and the saveCalibrationData command is issued, Bottlenose recalibrates itself and generates a calibration settings for each supported image profile. It is important to note that this operation cannot be performed while the camera is streaming.

Typing the calibration parameters one by one inside eBUS Player or any other framework is error-prone. The Bottlenose File utility can be used to directly upload a calibration file inside the camera for calibration.

Calibration parameters are lens dependent. This means the camera has to be re-calibrated if the lenses are changed or refocused. Bottlenose does not ship with any lens, but supports a wide variety of off-the-shelf lenses. Lenses should be selected based on the application requirements. More importantly, the selection of any lens should also be guided by the camera model supported by Bottlenose.

Image Undistortion and Rectification

Bottlenose is able to undistort and rectifiy images on-the-fly using the provided calibration data. These image processing functions can be turned on or off using the GigE Vision interface. Undistorted and rectified images are important for some image processing functions such as disparity and depth estimation.

Disparity Computation

Bottlenose stereo can generate a dense disparity map of a scene producing a fully color-aligned RGB-D image. Bottlenose implements the semi-global matching (SGM) algorithm for its disparity computation and therefore expects that images are rectified to produce accurate results. The disparity image can be converted into a depth map or a fully triangulated 3D point cloud.

Disparity Parameters

The disparity module has many parameters that can be tuned to affect how Bottlenose computes the disparity map. These parameters are located under the DisparityParameters section of the Device Control. User can then change values of parameters such as block width and height, minimum disparity, maximum disparity, disparity threshold, etc.

👍

Click on individual parameter to obtain a description and possible range of values.

Disparity Mode

Once calibrated, it is easy to put your camera into disparity mode. For this, under ImageFormatControl, change the PixelFormat from YUV422_8 to Coord3D_C16. Images are transported as YUV422 whereas Disprity data is transported as Coord3D_C16. This is nothing but an alias for the standard int16 data type. Disparity images can be visualized as a grey image inside your eBUSPlayer. The Bottlenose Calibrator utility can color code the disparity image.

👍

The disparity image is computed for the set image resolution. For a different resolution, stop streaming and change the Width and Height under ImageFormatControl.

The Bottlenose StereoViewer utility can also be used to stream color-coded disparity maps.

Disparity Alignment

Bottlenose aligns its disparity with its left image, that is the image produced by its left image sensor.