Stereo Vision

Includes stereo mechanism

Implement SGBM Algorithm

class py2vision.stereo.match_method.Matcher(strategy: py2vision.stereo.match_method.MatcherStrategy)

The Context defines the interface of interest to clients. The matcher accepts a strategy through the constructor, but also provides a setter to change it at runtime.

match()

The Context delegates some work to the Strategy object instead of implementing multiple versions of the algorithm on its own.

strategy

The Context maintains a reference to one of the Strategy objects. The Context does not know the concrete class of a strategy. It should work with all strategies via the Strategy interface.

class py2vision.stereo.match_method.MatcherStrategy

The Strategy interface declares operations common to all supported versions of some algorithm.

The Context uses this interface to call the algorithm defined by Concrete Strategies.

class py2vision.stereo.match_method.StereoSGBM(min_disp=0, max_disp=160, window_size=3, p1=216, p2=864, pre_filter_cap=63, mode=1, speckle_window_size=1100, speckle_range=1, uniqueness_ratio=5, disp_12_max_diff=-1)

To create an instance of stereo SGBM algorithm.

Parameters:

• min_disp – Minimum possible disparity value. Normally, it is zero but sometimes rectification algorithms can shift images, so this parameter needs to be adjusted accordingly.
• max_disp – Maximum disparity minus minimum disparity. The value is always greater than zero. In the current implementation, this parameter must be divisible by 16.
• window_size – Matched block size. It must be an odd number >=1 . Normally, it should be somewhere in the 3..11 range.
• p1 – The first parameter controlling the disparity smoothness.
• p2 – The second parameter controlling the disparity smoothness. The larger the values are, the smoother the disparity is. P1 is the penalty on the disparity change by plus or minus 1 between neighbor pixels. P2 is the penalty on the disparity change by more than 1 between neighbor pixels. The algorithm requires P2 > P1 . See stereo_match.cpp sample where some reasonably good P1 and P2 values are shown (like 8*number_of_image_channels*SADWindowSize*SADWindowSize and 32*number_of_image_channels*SADWindowSize*SADWindowSize, respectively).
• pre_filter_cap – Truncation value for the prefiltered image pixels. The algorithm first computes x-derivative at each pixel and clips its value by [-preFilterCap, preFilterCap] interval. The result values are passed to the Birchfield-Tomasi pixel cost function.
• mode – Set it to StereoSGBM_MODE_HH to run the full-scale two-pass dynamic programming algorithm. It will consume O(W*H*numDisparities) bytes, which is large for 640x480 stereo and huge for HD-size pictures. By default, it is set to false .
• speckle_window_size – Maximum size of smooth disparity regions to consider their noise speckles and invalidate. Set it to 0 to disable speckle filtering. Otherwise, set it somewhere in the 50-200 range.
• speckle_range – Maximum disparity variation within each connected component. If you do speckle filtering, set the parameter to a positive value, it will be implicitly multiplied by 16. Normally, 1 or 2 is good enough.
• uniqueness_ratio – Margin in percentage by which the best (minimum) computed cost function value should “win” the second best value to consider the found match correct. Normally, a value within the 5-15 range is good enough.
• disp_12_max_diff – Maximum allowed difference (in integer pixel units) in the left-right disparity check. Set it to a non-positive value to disable the check.

match()

Return stereo sgbm instance

Standard Stereo Builder

class py2vision.stereo.standard_stereo.StandardStereo(cam_left: py2vision.input_output.camera.Camera, cam_right: py2vision.input_output.camera.Camera, fish_eye=True)

An emulation of a real world stereo system with his relevant parameters, like fundamental matrix, rotation matrix, translation matrix and esential matrix.

camL

left camera instance.

camR

right camera instance.

fish_eye

A boolean if is true it will calibrate with cv2.fisheye.calibrate if no it’ll use normal calibration, fish eye is recomended when cameras has an field of view > 160.

calibrate(images_left_path, images_right_path, pattern_type='chessboard', pattern_size=(8, 5), show=True)

Compute intrinsics and extrinsics parameters for two cameras at once. Note: if you want a good calibration your images in ‘images_left_path’ and ‘images_right_path’ have to follow a format for example: example/left_images/photo_1.jpg and example/right_images/photo_1.jpg the number is important because calibrate method order the pair of images with sorted method.

Parameters:

• images_left_path – folder where is saved left calibration pattern photos.
• images_right_path – folder where is saved right calibration pattern photos.
• pattern_type – It can be “circles” pattern or “chessboard” pattern (default).
• pattern_size – If pattern_type is “chessboard” this the Number of inner corners per a chessboard row and column. But If pattern_type is “circles” this will be the number of circles per row and column.
• show – if is true it show corners or centers found by calibration algorithm at each iteration.

Returns:

Three floats, The first is the reprojection error in left side, the another one is right side and the third one is rms error (these have to be less than 1).

Raises:

OSError – If didn’t find photos on images_left_path or images_right_path folders.

get_stereo_maps(path)

Loads stereo maps parameters from a xml file

Parameters:path – path where is saved xml file. (Note: if you don’t have stereo maps you can use calibrate method first, then use rectify method and pass true in export file argument)

print_parameters(one_camera_parameters=['matrix', 'dist_coeffs'], stereo_parameters=['Q', 'rot', 'trans', 'e_matrix', 'f_matrix'])

Print required individual camera parameters and stereo parameters already defined.

Parameters:

• one_camera_parameters – All elements in list will be printed, they match with cameras parameters.
• stereo_parameters – All elements in list will be printed, they match with stereo parameters.

rectify(image_left__dims, image_right_dims, export_file_name='stereoMap', alpha=1, output_size=(0, 0), export_file=True)

Compute stereo rectification maps and export left and right stereo maps in xml format. Note: you will need to calibrate first

Parameters:

• image_left__dims – a tuple or list with left image (width, height).
• image_right_dims – a tuple or list with right image (width, height).
• export_file_name – personalize the name of output file.
• alpha – free scaling parameter. If it is -1 or absent, the function performs the default scaling. Otherwise, the parameter should be between 0 and 1. alpha=0 means that the rectified images are zoomed and shifted so that only valid pixels are visible (no black areas after rectification). alpha=1 (default) means that the rectified image is decimated and shifted so that all the pixels from the original images from the cameras are retained in the rectified images (no source image pixels are lost). Any intermediate value yields an intermediate result between those two extreme cases (Only apply for no fish eye cameras).
• output_size – New image resolution after rectification. When (0,0) is passed (default), it is set to the original image Size. Setting it to a larger value can help you preserve details in the original image, especially when there is a big radial distortion.
• export_file – if is true this method will save the parameters in an xml with the name in export_file_name.

Raises:

AttributeError – If haven’t calibrated before, you need all parameters of calibrate() method.

take_dual_photos(num_photos=15, save_dir_left='images_left', save_dir_right='images_right', prefix_name='photo')

A simple way to take photos in console and save in a folder.

Parameters:

• num_photos – number of photos to take, 15 by default.
• save_dir_left – directory name where the left photos will be saved.
• save_dir_right – directory name where the right photos will be saved.
• prefix_name – A prefix for the names of the photos.

Raises:

OSError – if folder exist.

class py2vision.stereo.standard_stereo.StandardStereoBuilder(cam_left: py2vision.input_output.camera.Camera, cam_right: py2vision.input_output.camera.Camera, stereo_maps_path)

Implement methods to get depth using OpenCV matchers like SGBM or BM.

camL

left camera instance.

camR

right camera instance.

stereo_maps_path

an xml with Stereo rectification maps

estimate_3D_points(points, disparity, Q)

Convert image plane points to homogeneous 3d points.

Parameters:

• points – contains the (x, y) coordinates in image plane to convert to (X, Y, Z).
• disparity – a disparity map with a shape of (w, h, 1)
• Q – a 4x4 array with the following structure, [[1 0 0 -cx ][0 1 0 -cy ][0 0 0 f ][0 0 -1/Tx (cx - cx’)/Tx ]] cx: is the principal point x in left image, cx’: is the principal point x in right image, cy: is the principal point y in left image, f: is the focal lenth in left image, Tx: The x coordinate in Translation matrix.

Returns:

An array of points in 3D homogeneous coordinates (X, Y, Z, W).

estimate_depth_map(disparity, Q, frame, metrics=True)

Convert disparity map to depth map.

Parameters:

• frame – left or right frame.
• disparity – a disparity map with a shape of (w, h, 1)
• Q – a 4x4 array with the following structure, [[1 0 0 -cx ][0 1 0 -cy ][0 0 0 f ][0 0 -1/Tx (cx - cx’)/Tx ]] cx: is the principal point x in left image, cx’: is the principal point x in right image, cy: is the principal point y in left image, f: is the focal lenth in left image, Tx: The x coordinate in Translation matrix.
• metrics – a boolean, if is true print by console the time of execution of depth map step.

Returns:

reprojected points image and a depth map in RGB

estimate_disparity_colormap(disparity)

It converts disparity maps with a shape of (w, h, 1) to (w, h, 3).

Parameters:disparity – a disparity map with a shape of (w, h, 1). Returns:disparity with color

find_epilines(frameL, frameR)

Draw epilines to both frames.

Parameters:

• frameL – it’s the left frame.
• frameR – it’s the right frame.

Returns:

Two elements, the left and right frame with epilines

get_product()

Return: StandardStereo builded object

match(frameL, frameR, matcher: py2vision.stereo.match_method.Matcher, metrics=True)

Apply stereo SGBM.

Parameters:

• frameL – it’s the left frame.
• frameR – it’s the right frame.
• metrics – a boolean, if is true print by console the time of execution of correspondence step.

Returns:

left and right disparity maps and even matcher instance.

post_process(frameL, left_disp, right_disp, matcher, lmbda=8000, sigma=1.5, metrics=True)

Apply wls filter in disparity maps to smooth contours.

Parameters:

• frameL – it’s the left frame.
• left_disp – it’s the left disparity frame.
• right_disp – it’s the right disparity frame.
• matcher – it’s the matcher instance.
• lmbda – is a parameter defining the amount of regularization during filtering. Larger values force filtered disparity map edges to adhere more to source image edges. Typical value is 8000. Only valid in post processing step.
• sigma – is a parameter defining how sensitive the filtering process is to source image edges. Large values can lead to disparity leakage through low-contrast edges. Small values can make the filter too sensitive to noise and textures in the source image. Typical values range from 0.8 to 2.0. Only valid in post processing step
• metrics – if is true print by console the time of execution of post process step.

Returns:

Improved disparity map

pre_process(frameL, frameR, downsample=2)

First, it transform from BGR to gray, next apply rectification and finally apply pyramid subsampling.

Parameters:

• frameL – it’s the left frame
• frameR – it’s the right frame
• downsample – if it is true, it will apply blurry in both frames and downsamples it. The downsampling factor just can be 2, 4, 8, 16, 32, 64. If downsample factor is 1 or None or False won’t apply downsampling.

Returns:

Both frames to apply stereo correspondence or apply more processing to the images.

reset(cam_left: py2vision.input_output.camera.Camera, cam_right: py2vision.input_output.camera.Camera, stereo_maps_path) → None

Initialitation of product

Stereo Controller

class py2vision.stereo.stereo_builder.StereoController

The StereoController is only responsible for executing the stereo steps in a particular sequence. It is helpful when producing products according to a specific order or configuration.

compute_3D_map(frameL, frameR, Q, matcher: py2vision.stereo.match_method.Matcher, downsample=2, lmbda=8000, sigma=1.5, post_process=True, metrics=True)

Apply the pre process step, next compute left and right disparity maps, then execute the post process with a wls filter to improve the final result, and finally compute depth map.

Parameters:

• frameL – it’s the left frame.
• frameR – it’s the right frame.
• Q – a 4x4 array with the following structure, [[1 0 0 -cx ][0 1 0 -cy ][0 0 0 f ][0 0 -1/Tx (cx - cx’)/Tx ]], cx: is the principal point x in left image, cx’: is the principal point x in right image, cy: is the principal point y in left image, f: is the focal lenth in left image, Tx: The x coordinate in Translation matrix.
• matcher – A Matcher instance.
• downsample – if it is true, it will apply blurry in both frames and downsamples it. The downsampling factor just can be 2, 4, 8, 16, 32, 64. If downsample factor is 1 or None or False won’t apply downsampling.
• lmbda – is a parameter defining the amount of regularization during filtering. Larger values force filtered disparity map edges to adhere more to source image edges. Typical value is 8000. Only valid in post processing step
• sigma – is a parameter defining how sensitive the filtering process is to source image edges. Large values can lead to disparity leakage through low-contrast edges. Small values can make the filter too sensitive to noise and textures in the source image. Typical values range from 0.8 to 2.0. Only valid in post processing step
• post_process – if is true apply post_process and return improved disparity map, otherwise return left disparity map without post processing.
• metrics – if is true print by console the time of execution of correspondence, post process and depth map step.

Returns:

reprojected points image, a depth map in RGB and correspondence method used

compute_3D_points(frameL, frameR, points, Q, matcher: py2vision.stereo.match_method.Matcher, downsample=2, lmbda=8000, sigma=1.5, post_process=True, metrics=True)

Apply the pre process step, next compute left and right disparity maps, then execute the post process with a wls filter to improve the final result, and finally compute 3D points for an input array that can be [[x1, y1], [x2, y2], [x3, y3], … [xn, yn]].

Parameters:

• frameL – it’s the left frame.
• frameR – it’s the right frame.
• points – contains the (x, y) coordinates in image plane to convert to (X, Y, Z)
• Q – a 4x4 array with the following structure, [[1 0 0 -cx ][0 1 0 -cy ][0 0 0 f ][0 0 -1/Tx (cx - cx’)/Tx ]], cx: is the principal point x in left image, cx’: is the principal point x in right image, cy: is the principal point y in left image, f: is the focal lenth in left image, Tx: The x coordinate in Translation matrix.
• matcher – A Matcher instance.
• downsample – if it is true, it will apply blurry in both frames and downsamples it. The downsampling factor just can be 2, 4, 8, 16, 32, 64. If downsample factor is 1 or None or False won’t apply downsampling.
• lmbda – is a parameter defining the amount of regularization during filtering. Larger values force filtered disparity map edges to adhere more to source image edges. Typical value is 8000. Only valid in post processing step
• sigma – is a parameter defining how sensitive the filtering process is to source image edges. Large values can lead to disparity leakage through low-contrast edges. Small values can make the filter too sensitive to noise and textures in the source image. Typical values range from 0.8 to 2.0. Only valid in post processing step
• post_process – if is true apply post_process and return improved disparity map, otherwise return left disparity map without post processing.
• metrics – if is true print by console the time of execution of correspondence, post process and depth map step.

Returns:

Two elements, an array of points in 3D homogeneous coordinates (X, Y, Z, W) and correspondence method used

compute_disparity(frameL, frameR, matcher: py2vision.stereo.match_method.Matcher, downsample=2, lmbda=8000, sigma=1.5, post_process=True, metrics=True)

Apply the pre process step, next compute left and right disparity maps, and finally execute the post process with a wls filter to improve the final result.

Parameters:

• frameL – it’s the left frame.
• frameR – it’s the right frame.
• matcher – A Matcher instance.
• downsample – if it is true, it will apply blurry in both frames and downsamples it. The downsampling factor just can be 2, 4, 8, 16, 32, 64. If downsample factor is 1 or None or False won’t apply downsampling.
• lmbda – is a parameter defining the amount of regularization during filtering. Larger values force filtered disparity map edges to adhere more to source image edges. Typical value is 8000. Only valid in post processing step
• sigma – is a parameter defining how sensitive the filtering process is to source image edges. Large values can lead to disparity leakage through low-contrast edges. Small values can make the filter too sensitive to noise and textures in the source image. Typical values range from 0.8 to 2.0. Only valid in post processing step
• post_process – if is true apply post_process and return improved disparity map, otherwise return left disparity map without post processing.
• metrics – if is true print by console the time of execution of correspondence and post process steps.

Returns:

Two elements, disparity map and correspondence method used.

compute_disparity_color_map(frameL, frameR, matcher: py2vision.stereo.match_method.Matcher, downsample=2, lmbda=8000, sigma=1.5, post_process=True, metrics=True)

Apply the pre process step, next compute left and right disparity maps, then execute the post process with a wls filter to improve the final result, and finally compute disparity map with color.

Parameters:

• frameL – it’s the left frame.
• frameR – it’s the right frame.
• matcher – A Matcher instance.
• downsample – if it is true, it will apply blurry in both frames and downsamples it. The downsampling factor just can be 2, 4, 8, 16, 32, 64. If downsample factor is 1 or None or False won’t apply downsampling.
• lmbda – is a parameter defining the amount of regularization during filtering. Larger values force filtered disparity map edges to adhere more to source image edges. Typical value is 8000. Only valid in post processing step
• sigma – is a parameter defining how sensitive the filtering process is to source image edges. Large values can lead to disparity leakage through low-contrast edges. Small values can make the filter too sensitive to noise and textures in the source image. Typical values range from 0.8 to 2.0. Only valid in post processing step.
• post_process – if is true apply post_process and return improved disparity map, otherwise return left disparity map without post processing.
• metrics – if is true print by console the time of execution of correspondence and post process steps.

Returns:

Two elements, disparity map (with 3 colors channels) and correspondence method used

get_epilines(frameL, frameR)

Draw epilines to both frames.

Parameters:

• frameL (arr) – it’s the left frame.
• frameR (arr) – it’s the right frame.

Returns:

Two elements, the left and right frame with epilines.

pre_process_step(frameL, frameR, downsample=2)

First, it transform from BGR to gray, next apply rectification and finally apply pyramid subsampling.

Parameters:

• frameL – it’s the left frame
• frameR – it’s the right frame
• downsample – if it is true, it will apply blurry in both frames and downsamples it. The downsampling factor just can be 2, 4, 8, 16, 32, 64. If downsample factor is 1 or None or False won’t apply downsampling.

Returns:

Both frames to apply stereo correspondence or apply more processing to the images.

class py2vision.stereo.stereo_builder.StereoSystemBuilder

The Builder interface specifies methods for stereo contoller

Examples

If you want to use an interactive code I recommend this tutorial: https://github.com/corvus96/PyTwoVision/blob/master/tutorials/1_stereo_tutorial.ipynb