Inputs and Outputs

Contains the classes that allow the input of data to the stereo and recognition system through external hardware such as cameras, video files or even live transmission via wifi, in addition to obtaining the intrinsic and extrinsic parameters of the physical medium that captured the images. On the other hand, its other function is to merge the capabilities of the recognition module and the stereo module with the VisionSystem class.

Inputs like

class py2vision.input_output.camera.Camera(id, source)

An emulation of a real world camera with his relevant parameters, like camera matrix, extrinsics and intrinsics parameters.

Parameters:

• id – A string to identify our camera.
• source – When you use webcam you need to put (int) 0, if you want to use videos or streaming, you will need to put his URL or path even you can put in a video file or a image path.

calibrate(images_path='', pattern_type='chessboard', pattern_size=(8, 5), export_file=True, show=True, fish_eye=True)

Compute camera parameters.

Parameters:

• images_path – folder where is saved calibration pattern photos.
• pattern_type – It can be “circles” pattern or “chessboard” pattern (default).
• pattern_size – If pattern_type is “chessboard” this the Number of inner corners per a chessboard row and column. But If pattern_type is “circles” this will be the number of circles per row and column.
• export_file – To export camera parameters on xml file.
• show – if is true it show corners or centers found by calibration algorithm at each iteration.
• fish_eye – A boolean if is true it will calibrate with cv2.fisheye.calibrate if no it’ll use normal calibration, fish eye is recomended when cameras has an field of view > 160.

Returns:

RMS calibration’s error.

Raises:

• OSError – If didn’t find photos on images_path folder.
• ValueError – If pattern_type is different of ‘chessboard’ or ‘circles’ or when
• fish_eye isn’t a boolean.

get_parameters(path)

Loads camera parameters from a xml file :param path: a path where is saved xml file. (Note: if you don’t have any parameters you can use calibrate method first and then pass true in export file argument)

Raises:OSError – when path is wrong.

take_photos(num_photos=15, save_dir='images', prefix_name='photo', rotate=0, resize=False, resize_dim=(640, 480))

A simple way to take photos in console and save in a folder.

Parameters:

• num_photos – Number of photos to take, 15 by default.
• save_dir – Directory name where the photos will be saved.
• prefix_name – A prefix for the names of the photos.
• rotate – rotate input image around his center, 0 grades by default.
• resize – To resize input image.
• resize_dim – resize input image dimensions (width, height), its default is (640, 480).

Raises:

OSError – if folder exist.

How to calibrate a single camera?

from py2vision.input_output.camera import Camera

fisheye_camera = Camera("fisheye", "A_PATH_OR_A_NAME_FOR_CALIBRATION_IT_DOESN'T_MATTER")
fisheye_camera.calibrate("A_CALIBRATION_IMAGE_FOLDER_PATH", show=False, export_file=False)

Outputs like

class py2vision.input_output.vision_system.VisionSystem(cam_left: py2vision.input_output.camera.Camera, cam_right: py2vision.input_output.camera.Camera, stereo_maps_path, matcher: py2vision.stereo.match_method.Matcher, q_matrix)

Provide an interface to apply recognition and stereo vision. Initialization of all necessary parameteres to implement an stereo-recognition system

Parameters:

• cam_left – a camera instance which can be streaming, mp4 file path, image path or even realtime source.
• cam_right – a camera instance which can be streaming, mp4 file path, image path or even realtime source.
• stereo_maps_path – a path stereo maps with rectify images.
• q_matrix – a 4x4 array with the following structure, [[1 0 0 -cx ][0 1 0 -cy ][0 0 0 f ][0 0 -1/Tx (cx - cx’)/Tx ]] cx: is the principal point x in left image cx’: is the principal point x in right image cy: is the principal point y in left image f: is the focal lenth in left image Tx: The x coordinate in Translation matrix

image_pipeline(model, class_file_name, output_path='', input_size=416, score_threshold=0.3, iou_threshold=0.45, rectangle_colors='', nms_method='nms', post_process_match=True, lmbda=128.0, sigma=1.5, downsample_for_match=2, show_window=True, otsu_thresh_inverse=True, text_colors=(255, 255, 0))

Implement a stereo recognition system for images.

Parameters:

• model – expects a tensorflow model trained.
• class_file_name – it’s the path of classes .txt file
• output_path – if is an empty string, it won’t be saved, but it is a path it save like a video.
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.
• post_process_match – if is true apply post_process and return an improved disparity map, otherwise return left disparity map without post processing.
• lmbda – is a parameter defining the amount of regularization during filtering. Larger values force filtered disparity map edges to adhere more to source image edges. Typical value is 8000. Only valid in post processing step
• sigma – is a parameter defining how sensitive the filtering process is to source image edges. Large values can lead to disparity leakage through low-contrast edges. Small values can make the filter too sensitive to noise and textures in the source image. Typical values range from 0.8 to 2.0. Only valid in post processing step.
• downsample_for_match – if true, will apply the blur on both frames and demultiply it. The downsampling factor can be 2, 4, 8, 16, 32, 64. If the downsample factor is 1 or None or False it will not apply the downsampling.
• show_window – shows a window with the application.
• otsu_thresh_inverse – The Otsu threshold transforms a grayscale image into a binary image, if this variable is True the binary image will favor darker pixels otherwise it will favor lighter pixels.
• text_colors – a tuple that represents (R, G, B) colors for drawed text.

Returns:

an image processed, bounding boxes and 3D points.

Raises:

ValueError – if input images are not some of this formats [“.bmp”, “.dib”, “.jpg”, “.jpeg”, “.jpe”, “.png”, “.webp”].

realtime_or_video_pipeline(model, class_file_name, output_path='', input_size=416, score_threshold=0.3, iou_threshold=0.45, rectangle_colors='', nms_method='nms', post_process_match=True, lmbda=128.0, sigma=1.5, downsample_for_match=2, show_window=True, otsu_thresh_inverse=True, text_colors=(255, 255, 0))

Implement a stereo recognition system for video or streaming

Parameters:

• model – expects a tensorflow model trained.
• class_file_name – it’s the path of classes .txt file
• output_path – if is an empty string, it won’t be saved, but it is a path it save like a video.
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.
• post_process_match – if is true apply post_process and return an improved disparity map, otherwise return left disparity map without post processing.
• lmbda – is a parameter defining the amount of regularization during filtering. Larger values force filtered disparity map edges to adhere more to source image edges. Typical value is 8000. Only valid in post processing step
• sigma – is a parameter defining how sensitive the filtering process is to source image edges. Large values can lead to disparity leakage through low-contrast edges. Small values can make the filter too sensitive to noise and textures in the source image. Typical values range from 0.8 to 2.0. Only valid in post processing step.
• downsample_for_match – if true, will apply the blur on both frames and demultiply it. The downsampling factor can be 2, 4, 8, 16, 32, 64. If the downsample factor is 1 or None or False it will not apply the downsampling.
• show_window – shows a window with the application.
• otsu_thresh_inverse – The Otsu threshold transforms a grayscale image into a binary image, if this variable is True the binary image will favor darker pixels otherwise it will favor lighter pixels.
• text_colors – a tuple that represents (R, G, B) colors for drawed text.

How to implement a position system?

import wget
import os

from py2vision.input_output.vision_system import VisionSystem
from py2vision.input_output.camera import Camera
from py2vision.stereo.standard_stereo import StandardStereo
from py2vision.stereo.match_method import Matcher, StereoSGBM
from py2vision.recognition.yolov3_detector import ObjectDetectorYoloV3
from py2vision.recognition.selector import Recognizer

# You can get coco.names here https://github.com/pjreddie/darknet/blob/master/data/coco.names
classes_file = "coco.names"
work_dir = "work_dir"
stereo_maps_path = "stereoMap"

try:
    os.mkdir(work_dir)
except:
    pass

left_camera = Camera("left_camera", "left/left_indoor_photo_5.png")
right_camera = Camera("right_camera", "right/right_indoor_photo_5.png")
stereo_pair_fisheye = StandardStereo(left_camera, right_camera)
stereo_pair_fisheye.calibrate("left_camera_calibration_folder", "right_camera_calibration_folder", show=False)
stereo_pair_fisheye.rectify((640, 720), (640, 720), export_file=True, export_file_name=stereo_maps_path)
# Add path format
stereo_maps_path = stereo_maps_path + ".xml"
sgbm = StereoSGBM(min_disp=-32, max_disp=32, window_size=3, p1=107, p2=710, pre_filter_cap=36, speckle_window_size=117, speckle_range=5, uniqueness_ratio=3, disp_12_max_diff=-38)
matcher = Matcher(sgbm)
lmbda = 13673
sigma = 1.3175
yolov3 = ObjectDetectorYoloV3("test", 80, training=False)
recognizer = Recognizer(yolov3)
link_yolov3_weights = "https://pjreddie.com/media/files/yolov3.weights"
wget.download(link_yolov3_weights)
weights_file = os.path.basename(link_yolov3_weights)
recognizer.restore_weights(weights_file)
model = recognizer.get_model()
vis_sys = VisionSystem(left_camera, right_camera, stereo_maps_path, matcher, stereo_pair_fisheye.Q)
# Here the magic happens
vis_sys.image_pipeline(model, classes_file, os.path.join(work_dir, "test_position.jpg"),  lmbda=lmbda, sigma=sigma, downsample_for_match=None, show_window=True, score_threshold=0.5, iou_threshold=0.6, otsu_thresh_inverse=True, text_colors=(0, 0, 0))