How do I speed up the YOLO v3-v4 inferencing?

Question

I run YOLO V3 or YOLO v4 on Jeston Xavier, DNN OpenCV version, It is very slow barely reaches ~ 7- on how to speed up inference?

Answer 1

Two things you could try to speed up inference:

• Use a smaller network size. Use yolov4-416 instead of yolov4-608 for example. This does probably come at the cost of lower accuracy.
• Try converting your network to TensorRT and use mixed precision (FP16 will give a huge performance increase and INT8 even more although then you have to recallibrate your network)

For the last one I would advise following this excellent blog:

blog

Answer 2

You can reduce the image size in the .cfg file:

width=416
height=416

• Use multiples of 32. (This can affect your accuracy if the objects in your scene are too small).
• And you can crop your input image to keep only the area of interest in a square shape. This helps to keep any small objects from getting compressed too much when you reduce the image size as Yolo will compress your input image to the above width and height while maintaining the image ratio.

You can also try Yolov3-tiny or Yolov4-tiny, which is a smaller network but this also might affect your accuracy. (These solutions have been based on my experience. There is always a trade-off between speed and accuracy when dealing with Yolo or other convolutional networks.)