Object detection (Recognition)

Encompasses training, inference, printing the network structure, evaluating it and recovering the pre-trained weights.

Networks selector

class py2vision.recognition.selector.NeuralNetwork

The Implementation defines the interface for all implementation classes. It doesn’t have to match the Abstraction’s interface. In fact, the two interfaces can be entirely different. Typically the Implementation interface provides only primitive operations, while the Abstraction defines higher-level operations based on those primitives.

class py2vision.recognition.selector.Recognizer(neural_network: py2vision.recognition.selector.NeuralNetwork)

An Abstraction that expects an implementation of a detector like: ‘ObjectDetectorYoloV3’.

evaluate(model, dataset, classes_file, score_threshold=0.05, iou_threshold=0.5, test_input_size=416)

Apply evaluation using mAP.

Parameters:

• model – a tesorflow detection model.
• dataset – an YoloV3DatasetGenerator instance with test dataset.
• classes_file – a string corresponding to the classes file (a .txt file with a list of classes) is located.
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm.
• test_input_size – integer to resize an input image from their original dimensions to an square image.

Returns:

mAP score

get_model()

Returns a tensorflow model with an implemented network

inference(image_path, input_size=416, score_threshold=0.3, iou_threshold=0.45, nms_method='nms')

Apply inference with trained model.

Parameters:

• image_path – a path to an image.
• input_size – integer to resize an input image from their original dimensions to an square image.
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.

Returns:

An array with bounding boxes.

print_model()

Print network summary for debugging purposes.

restore_weights(weights_file, use_checkpoint=False)

Load previously trained model weights.

Parameters:

• weights_file – beginning by project root this is the path where is save your weights; example: “weights/weights_01.h5”.
• use_checkpoint – if you wanna use a .ckpt file this variable should be True.

train(train_annotations_path, test_annotations_path, class_file_name, checkpoint_path='checkpoints', use_checkpoint=False, warmup_epochs=2, epochs=100, log_dir='logs', save_only_best_model=True, save_all_checkpoints=False, batch_size=4, lr_init=0.0001, lr_end=1e-06, strides=[8, 16, 32], anchors=[[[10, 13], [16, 30], [33, 23]], [[30, 61], [62, 45], [59, 119]], [[116, 90], [156, 198], [373, 326]]], anchor_per_scale=3, max_bbox_per_scale=100)

Train an ssd network.

Parameters:

• train_annotations_path – a string corresponding to the folder where train annotations are located.
• test_annotations_path – a string corresponding to the folder where test annotations are located.
• class_file_name – a string corresponding to the classes file (a .txt file with a list of classes) is located.
• checkpoint_path – a string corresponding to the checkpoint file that is inside of a checkpoints folder.
• use_checkpoint – a boolean that controls if use chepoint before train.
• warmup_epochs – an hiperparameter that update learning rate like this paper https://arxiv.org/pdf/1812.01187.pdf&usg=ALkJrhglKOPDjNt6SHGbphTHyMcT0cuMJg.
• epochs – Number of epochs to train.
• log_dir – a folder to save logs.
• save_only_best_model – if is true the model will be saved when best validation loss > total validation loss/total test elements, but if it isn’t true model will be saved always.
• save_all_checkpoints – it is a boolean, if is true model will be saved in each epoch.
• batch_size – an integer with the size of batches in test and train datasets.
• lr_init – a float which is initial learning rate.
• lr_end – a float which is final learning rate.
• strides – a list with the strides in a yolo model.
• anchors – these are the yolo anchors sizes.
• anchor_per_scale – an integer with the number of anchor boxes per scale.
• max_bbox_per_scale – nan integer with the number of bounding boxes per scale.

train_using_weights(train_annotations_path, test_annotations_path, class_file_name, weights_path, checkpoint_path='checkpoints', use_checkpoint=False, warmup_epochs=2, epochs=100, log_dir='logs', save_only_best_model=True, save_all_checkpoints=False, batch_size=4, lr_init=0.0001, lr_end=1e-06, strides=[8, 16, 32], anchors=[[[10, 13], [16, 30], [33, 23]], [[30, 61], [62, 45], [59, 119]], [[116, 90], [156, 198], [373, 326]]], anchor_per_scale=3, max_bbox_per_scale=100)

Train with transfer learning.

Parameters:

• train_annotations_path – a string corresponding to the folder where train annotations are located.
• test_annotations_path – a string corresponding to the folder where test annotations are located.
• class_file_name – a string corresponding to the classes file (a .txt file with a list of classes) is located.
• weights_path – a path which in case if it’s an url with weights like: ‘https://pjreddie.com/media/files/yolov3.weights’ or ‘https://pjreddie.com/media/files/yolov3-tiny.weights’ this method first download the weights and then train the net but if the path is a local file the method is going to load the weights and next train the network.
• checkpoint_path – a string corresponding to the checkpoint file that is inside of a checkpoints folder.
• use_checkpoint – a boolean that controls if use chepoint before train.
• warmup_epochs – an hiperparameter that update learning rate like this paper https://arxiv.org/pdf/1812.01187.pdf&usg=ALkJrhglKOPDjNt6SHGbphTHyMcT0cuMJg.
• epochs – Number of epochs to train.
• log_dir – a folder to save logs.
• save_only_best_model – if is true the model will be saved when best validation loss > total validation loss/total test elements, but if it isn’t true model will be saved always.
• save_all_checkpoints – it is a boolean, if is true model will be saved in each epoch.
• batch_size – an integer with the size of batches in test and train datasets.
• lr_init – a float which is initial learning rate.
• lr_end – a float which is final learning rate.
• strides – a list with the strides in a yolo model.
• anchors – these are the yolo anchors sizes.
• anchor_per_scale – an integer with the number of anchor boxes per scale.
• max_bbox_per_scale – nan integer with the number of bounding boxes per scale.

YOLOV3 implementation

class py2vision.recognition.yolov3_detector.ObjectDetectorYoloV3(model_name, num_class, input_shape=[416, 416, 3], version='yolov3', training=False, gpu_name=None)

Made of an Yolo network model and a dataset generator.

Parameters:

• mode_name – an string to naming the model.
• num_class – an integer with the numbers of classes in the model.
• input_shape – A tuple with dims shape (height, weight, channels).
• version – it can be ‘yolov3’ or ‘yolov3_tiny’.
• training – a boolean that change depending if you want to train the model
• gpu_name – a gpu name if it is None this class search automatically a gpu compatible.

model

A model instance.

num_class

an integer with the numbers of classes in the model.

version

it can be ‘yolov3’ or ‘yolov3_tiny’.

model_name

an string to naming the model.

input_shape

A tuple with dims shape (height, weight, channels).

gpus

a list with all allowed gpus.

conv_tensors

these are the ouput of build yolov3 without prediction layer.

build_model(conv_tensors, training)

Build the complete yolo model and return model instance.

Parameters:

• conv_tensors – a tensor with convolutional layers of a yolo network without output layers or prediction layers.
• training – a boolean that change network structure, if is true the last layers will be predict tensors otherwise it will be output tensors.

Returns:

A yolo model.

evaluate(model, dataset, classes_file, score_threshold=0.05, iou_threshold=0.5, test_input_size=416)

Apply evaluation using mAP.

Parameters:

• model – a tesorflow detection model.
• dataset – an YoloV3DatasetGenerator instance with test dataset.
• classes_file – a string corresponding to the classes file (a .txt file with a list of classes) is located.
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm.
• test_input_size – integer to resize an input image from their original dimensions to an square image.

Returns:

mAP score

inference(image_path, input_size=416, score_threshold=0.3, iou_threshold=0.45, nms_method='nms')

Apply inference with trained model.

Parameters:

• image_path – a path to an image.
• input_size – integer to resize an input image from their original dimensions to an square image.
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.

Returns:

An array with bounding boxes

print_summary()

Print network summary for debugging purposes.

restore_weights(weights_file, use_checkpoint=False)

Load previously trained model weights.

Parameters:

• weights_file – beginning by project root this is the path where is save your weights; example: “weights/weights_01.h5”.
• use_checkpoint – if you wanna use a .ckpt file this variable should be True.

train(train_annotations_path, test_annotations_path, class_file_name, checkpoint_path='checkpoints', use_checkpoint=False, warmup_epochs=2, epochs=100, log_dir='logs', save_only_best_model=True, save_all_checkpoints=False, batch_size=4, lr_init=0.0001, lr_end=1e-06, strides=[8, 16, 32], anchors=[[[10, 13], [16, 30], [33, 23]], [[30, 61], [62, 45], [59, 119]], [[116, 90], [156, 198], [373, 326]]], anchor_per_scale=3, max_bbox_per_scale=100)

Train an yolov3 network or yolov3 tiny.

Parameters:

• train_annotations_path – a string corresponding to the folder where train annotations are located.
• test_annotations_path – a string corresponding to the folder where test annotations are located.
• class_file_name – a string corresponding to the classes file (a .txt file with a list of classes) is located.
• checkpoint_path – a string corresponding to the checkpoint file that is inside of a checkpoints folder.
• use_checkpoint – a boolean that controls if use chepoint before train
• warmup_epochs – an hiperparameter that update learning rate like this paper https://arxiv.org/pdf/1812.01187.pdf&usg=ALkJrhglKOPDjNt6SHGbphTHyMcT0cuMJg
• epochs – Number of epochs to train.
• log_dir – a folder to save logs.
• save_only_best_model – if is true the model will be saved when best validation loss > total validation loss/total test elements, but if it isn’t true model will be saved always.
• save_all_checkpoints – it is a boolean, if is true model will be saved in each epoch.
• batch_size – an integer with the size of batches in test and train datasets.
• lr_init – a float which is initial learning rate
• lr_end – a float which is final learning rate
• strides – a list with the strides in a yolo model.
• anchors – these are the yolo anchors sizes.
• anchor_per_scale – an integer with the number of anchor boxes per scale.
• max_bbox_per_scale – nan integer with the number of bounding boxes per scale.

train_step(image_data, target, optimizer, lr_init=0.0001, lr_end=1e-06)

training step.

Parameters:

• image_data – an image.
• target – labels
• optimizer – an tensorflow optimizer like Adams optimizer.
• lr_init – initial leraning rate hiperparameter.
• lr_end – final learning rate hiperparameter.

Returns:

(global_steps, optimizer.lr, giou_loss, conf_loss, prob_loss, total_loss)

validate_step(image_data, target)

Validation step.

Parameters:

• image_data – an image.
• target – labels.

Returns:

(giou_loss, conf_loss, prob_loss, total_loss)

Inference modes

class py2vision.recognition.detection_mode.DetectImage

detect(model, input_path, class_file_name, output_path='', input_size=416, score_threshold=0.3, iou_threshold=0.45, rectangle_colors='', nms_method='nms', show=True)

Apply detection pipeline in an image, if you want to close the window just press ‘q’.

Parameters:

• model – expects a tensorflow model trained.
• input_path – an image path
• class_file_name – it’s the path of classes .txt file
• output_path – if is an empty string, it won’t be saved, but it is a path it save like an image.
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.
• show – a boolean to show frame pcessed.

Returns:

an image with prediction drawed

prepare_input(image_path)

Get a path an convert in an image array.

Parameters:image_path – a path. Returns:An image array

show(image, output_path='')

Get a an image, then save it and finally show it.

Parameters:

• image – an image array.
• output_path – if is an empty string, it won’t be saved, but it is a path it save like an image.

class py2vision.recognition.detection_mode.DetectRealTime

detect(model, camera: py2vision.input_output.camera.Camera, class_file_name, output_path='', input_size=416, score_threshold=0.3, iou_threshold=0.45, rectangle_colors='', nms_method='nms', show=True)

Apply detection pipeline in realtime, if you want to close the window just press ‘q’.

Parameters:

• model – expects a tensorflow model trained.
• camera – An instance of camera object.
• class_file_name – it’s the path of classes .txt file.
• output_path – if is an empty string, it won’t be saved, but it is a path it save like a video.
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm.
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.
• show – a boolean to show frame pcessed.

prepare_input(vid, output_path)

Initialization for writing and reading videos

Parameters:

• vid – a cv.VideoCapture instance.
• output_path – if is an empty string, it won’t be saved, but it is a path it save like an image.

Returns:

a tuple where its first argument is an instance to Opencv video writes, the second element is an instance to read frames, and the last element is the frames per second

show(image, camera: py2vision.input_output.camera.Camera)

Show an image

Parameters:

• image – an image array.
• camera – An instance of camera object.

class py2vision.recognition.detection_mode.DetectRealTimeMP

detect(model, camera: py2vision.input_output.camera.Camera, class_file_name, output_path='', input_size=416, score_threshold=0.3, iou_threshold=0.45, rectangle_colors='', nms_method='nms')

Apply detection pipeline using multiprocessing, if you want to close the window just press ‘q’.

Parameters:

• model – expects a tensorflow model trained.
• camera – An instance of camera object.
• class_file_name – it’s the path of classes .txt file
• output_path – if is an empty string, it won’t be saved, but it is a path it save like a video.
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.

multi_process_initialization(model, original_frames, frames_data, predicted_data, processed_frames, processing_times, final_frames, class_file_name, input_size=416, score_threshold=0.3, iou_threshold=0.45, rectangle_colors='', nms_method='nms')

Apply an initialite multi processing prediction, postprocessing and show steps.

Parameters:

• model – expects a tensorflow model trained.
• original_frames – a queue from multiprocessing package, that corresponds with frames.
• frames_data – a queue from multiprocessing package, that corresponds with frames.
• predicted_data – a queue from multiprocessing package, that corresponds with predictions.
• processed_frames – a queue from multiprocessing package, that corresponds with processed frames.
• times (processing) – a queue from multiprocessing package, that corresponds with times.
• final_frames – a queue from multiprocessing package, that corresponds with final frames.
• class_file_name – it’s the path of classes .txt file
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.

Returns:

an Process tuple (prediction, postprocessing, show)

postprocess_mp(predicted_data, original_frames, processed_frames, processing_times, input_size, class_file_name, score_threshold, iou_threshold, rectangle_colors, nms_method)

Improve bounding boxes using multiprocessing. It needs to be initialized.

Parameters:

• predicted_data – a queue from multiprocessing package, that corresponds with predictions.
• original_frames – a queue from multiprocessing package, that corresponds with frames.
• processed_frames – a queue from multiprocessing package, that corresponds with processed frames.
• times (processing) – a queue from multiprocessing package, that corresponds with times.
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• class_file_name – it’s the path of classes .txt file
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.

predict_bbox_mp(model, frames_data, predicted_data, processing_times)

predict bounding boxes using multiprocessing. It needs to be initialized.

Parameters:

• model – expects a tensorflow model trained.
• frames_data – a queue from multiprocessing package, that corresponds with frames.
• predicted_data – a queue from multiprocessing package, that corresponds with predictions.
• processing_times – a queue from multiprocessing package, that corresponds with times.

prepare_input(vid, output_path) → None

Initialization for writing and reading videos.

Parameters:

• vid – cv.VideoCapture instance
• output_path – if is an empty string, it won’t be saved, but it is a path it save like an image.

Returns:

a tuple where its first argument is an instance to Opencv video writes, the second element is an instance to read frames, and the last element is the frames per second

show(processed_frames, final_frames)

Show preocessed input.

Parameters:

• processed_frames – a queue from multiprocessing package, that corresponds with processed frames.
• final_frames – a queue from multiprocessing package, that corresponds with final frames.

class py2vision.recognition.detection_mode.DetectVideo

detect(model, input_path, class_file_name, output_path='', input_size=416, score_threshold=0.3, iou_threshold=0.45, rectangle_colors='', nms_method='nms', show=True)

Apply detection pipeline in a saved video, if you want to close the window just press ‘q’.

Parameters:

• model – expects a tensorflow model trained.
• input_path – a video path.
• class_file_name – it’s the path of classes .txt file
• output_path – if is an empty string, it won’t be saved, but it is a path it save like a video.
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.
• show – a boolean to show frame pcessed

prepare_input(input_path, output_path)

Initialization for writing and reading videos.

Parameters:

• input_path – an video path.
• output_path – if is an empty string, it won’t be saved, but it is a path it save like an image.

Returns:

a tuple where its first argument is an instance to Opencv video writes, the second element is an instance to read frames, and the last element is the frames per second

show(image)

Show an image. :param image: an image array.

class py2vision.recognition.detection_mode.DetectionMode

The Abstract Class which defines detect method that contains the skeleton of detection algorithm with different methods such as detection on images, detection on video, detection using multiprocessing and real time detection.

camera_input(camera: py2vision.input_output.camera.Camera)

If camera source is webcam or other it is gonna create attribute ‘vid’, otherwise return a frame from streaming.

Parameters:camera – is an instance of Camera Object

detect()

Apply detection pipeline

draw(original_image, bboxes, class_file_name, rectangle_colors, homogeneous_points=None, text_colors=(255, 255, 0))

Draw bounding boxes on images.

Parameters:

• original_image – an array which correspond with an image
• bboxes – their bounding boxes.
• class_file_name – a path with a .txt file where the classes are saved.
• rectangle_colors – if this parameter is a string empty bounding box colors will be assing by default, however if rectangle_colors is a tuple like: (R, G, B) that will be bounding box colors.
• homogeneous_points – an array with dimensions n x 4 where each row is like (X, Y, Z, W). However if is None it won’t be drawed.

Returns:

An image with bounding boxes and homogeneous coordinates.

postprocess_boxes(original_image, pred_bbox, input_size=416, score_threshold=0.3, iou_threshold=0.45, nms_method='nms')

Apply a postprocess and non maximum supression step and resize bboxes.

Parameters:

• original_image – expects a tensorflow model trained.
• pred_bbox – a tensor of predicted bounding boxes.
• input_size – integer to resize bounding boxes from their resized dimensions to original dimensions (input_size).
• score_threshold – if the score of a bounding boxes is less than score_threshold, it will be discard.
• iou_threshold – a parameter between (0, 1) which is used for nms algorithm
• nms_method – a string that can be ‘nms’ or ‘soft-nms’.

Returns:

Improved bounding boxes

pre_process(image, input_size)

Apply resize images step.

Parameters:

• image – expects a an array.
• input_size – integer to resize an input image from their original dimensions to an square image.

Returns:

Resized images and bounding boxes.

predict(model, image_data)

Apply a prediction step.

Parameters:

• model – expects a tensorflow model trained.
• image_data – expects an array or tensor with image data.

Returns:

predicted bounding boxes.

prepare_input()

It’s the first step to convert inputs like video paths or images in compatible data.

show()

Show actual frame in a window

Examples

If you want to use an interactive code I recommend this tutorial: https://github.com/corvus96/PyTwoVision/blob/master/tutorials/2_YOLO_tutorial.ipynb