Deploying the Hand Detector onto Jetson TX2

Sep 25, 2018

In previous posts, I’ve shared how to apply TF-TRT to optimize pretrained object detection models, as well as how to train a hand detector with TensorFlow Object Detection API. Let’s now put things together by optimizing and deploying the hand detector on Jetson TX2.

Prerequisite

Follow my previous post, TensorFlow/TensorRT Models on Jetson TX2. Make sure your Jetson TX2 has been set up and could run camera_tf_trt.py successfully.

Note that I’ve just fixed a couple of bugs, as well as added support for the ‘ssd_mobilenet_v1_egohands’ model, in my jkjung-avt/tf_trt_models repo. So if you have cloned the repository previously, do pull the latest code from GitHub again.
Follow my other post, Training a Hand Detector with TensorFlow Object Detection API. Make sure you have successfully trained the ‘ssd_mobilenet_v1_egohands’ model. We would be using the checkpoint (saved model weights) file for the demonstration below.

Changes needed from previous tf_trt_model

The following description refers to code in tf_trt_models.

tf_trt_models would need the config and checkpoint files of the ‘ssd_mobilenet_v1_egohands’ model, to be able to compile an optimized tensorflow graph for inferencing. I made a copy of data/ssd_mobilenet_v1_egohands.config from the hand-detection-tutorial repo, while changing the score_threshold value from 1e-8 to 0.3 within. However, I did not put a copy of the model checkpoint files in the repository, since those files were too big. Please copy them over as instructed below.
In utils/ssd_util.py, I added get_egohands_model() for locating the model config and checkpoint files.
I made a copy of data/egohands_label_map.pbtxt. This prototxt file is used for converting class IDs to class names.

Running TF-TRT optimized ssd_mobilenet_v1_egohands model on Jetson TX2

Copy your ‘ssd_mobilenet_v1_egohands’ checkpoint files from your hand-detection-tutorial to your tf_trt_models directory. For example, I have both project code checked out under my ~/project/ folder, so I’d do the following. Note that we have trained the model for a total of 20,000 steps. That’s why the checkpoint files are marked with that number.

$ cd ~/project/tf_trt_models
$ cp ~/project/hand-detection-tutorial/ssd_mobilenet_v1_egohands/model.ckpt-20000.* ./data/ssd_mobilenet_v1_egohands/

When inferencing the model with camera_tf_trt.py, you’d need to specify the label map file and set number of classes to 1. Refer to the commands below for details.

Example #1: build TensorRT optimized ‘ssd_mobilenet_v1_egohands’ model and test with a still image. I took a picture of my son’s hands along with his newly assembled LEGO car for testing. The result looked good. camera_tf_trt.py ran at ~24.5 fps in this case.

$ python3 camera_tf_trt.py --image \
                           --filename jk-son-hands.jpg \
                           --model ssd_mobilenet_v1_egohands \
                           --labelmap data/egohands_label_map.pbtxt \
                           --num-classes 1 \
                           --build

Example #2: verify the optimized ‘ssd_mobilenet_v1_egohands’ model with live video feed from a USB webcam. Here I myself was acting as the model in front of the webcam :-)

$ python3 camera_tf_trt.py --usb \
                           --model ssd_mobilenet_v1_egohands \
                           --labelmap data/egohands_label_map.pbtxt \
                           --num-classes 1

Tested with my own hand in front of a USB webcam

In addition to image file and USB webcam, the camera_tf_trt.py script also supports video file, IP CAM, or Jetson onboard camera as input source. I’m not going to re-iterate those here. You can refer to --help message for more details.

Deploying the Hand Detector onto Jetson TX2

Deploying the Hand Detector onto Jetson TX2

Prerequisite

Changes needed from previous tf_trt_model

Running TF-TRT optimized ssd_mobilenet_v1_egohands model on Jetson TX2

Recommend

How I built TensorFlow 1.8.0 on Jetson TX2

TensorFlow/TensorRT Models on Jetson TX2

YOLOv3 on Jetson TX2

Building and Testing 'openalpr' on Jetson TX2

Measuring Caffe Model Inference Speed on Jetson TX2

Faster R-CNN on Jetson TX2

Single Shot MultiBox Detector (SSD) on Jetson TX2

YOLOv2 on Jetson TX2

Trying out TensorRT on Jetson TX2

Deep Learning Cats Dogs Tutorial on Jetson TX2

About Joyk