Turi Create: slow training performance on Blackmagic eGPU

332 Views Asked by At

On my Macbook Pro 13" I have the Blackmagic eGPU (AMD Radeon Pro 580) connected via USB-C. This should theoretically speed up my model training with Turi Create enormously.

For a small model in my case 15 labeled images (4k x 3k) and 500 iterations are used, which which take about 2 hours including the eGPU. Only CPU takes 4h, so the GPU speeds up, but not extremely.

In the Guide to Turi Create there is said that an object detection model with ~700 images and 4000 iterations is processed in 1 hour. So way faster.

While using CreateML I observe an increase of performance of at least 5x for transfer learning during the feature detection phase when using the eGPU.

Is this a problem of the framework itself?

Can I optimize the data or training parameters for better usage of the eGPU?

Is the data too small or the resolution too big to have optimal GPU usage over USB-C?

Class                                    : ObjectDetector

Schema
------
Model                                    : darknet-yolo
Number of classes                        : 4
Non-maximum suppression threshold        : 0.45
Input image shape                        : (3, 416, 416)

Training summary
----------------
Training time                            : 1h 29m 8s
Training epochs                          : 1066
Training iterations                      : 500
Number of examples (images)              : 15
Number of bounding boxes (instances)     : 49
Final loss (specific to model)           : 1.808
1

There are 1 best solutions below

2
On

It is the image size/resolution (4k x 3k) which creates the bottleneck to the GPU. Scaling the images down (and setting the labels accordingly) gets full speed of the eGPU (100x vs CPU).