Qualcomm SNPE block the UI Thread

Question

using the Qualcomm NeturalNetwork SDK. i can run the SNPE SDK Example and change to inception_v3 model, works fine.

but snpe will block-ui thread in the execute();

i have no way to stop this. android user will get bad UX.

i have tryed: low priority thread, job scheduler, etc

when i execute snpe with GPU, it always block the UI.

How can i config the SNPE, the Android UI is the high priority, SNPE is the lower priority, so we can get result quickly and do not block the UI

thank you.

Answer 1

The bulk operation on GPU blocks the rendering of new frames. It's hard to solve this problem and it actually have nothing to do with SNPE, because we can reproduce this issue using non-SNPE implementation (in-house OpenCL-based framework). You can simply change the placement of tensor operations to mitigate this problem. For example, you can do the computation on CPU (e.g.: tensorflow mobile), and the UI can be rendered properly while being much slower and CPU-hunger.

It's possible to visualize my explanation by on-device developer options. For more information, follow this link: https://developer.android.com/studio/profile/inspect-gpu-rendering#profile_rendering. You'll be able to see that several "Swap Buffer"1 operations could take unusually long intervals.

The best solution is to do computation on DSP with quantized network, but there are many limitations on the available operators and memory.

It's possible that Android 8.1 could solve these issues with NN-API abstraction and OS-level scheduling of GPU-resource, but I would not expect too much from Google.

BTW: I have a hypothetical scheme to mitigate this issue by fragmenting the bulk operations. In theory, if the worker-thread would sleep for 20ms between sub-50ms operations so that UI thread could render properly, the user experience should be tolerable since the FPS could be maintained above 15. We'll try this scheme because this handicapped scheme should still be much faster than schemes based on CPU.

Answer 2

For SNPE GPU runtime, you can use low execution hint, which will set SNPE to lowest GPU priority.

By setting ExecutionPriorityHint_t::Low thru SNPEBuilder::setExecutionPriorityHint()

Answer 3

You should be able to use an AsyncTask to run your inference on a background thread. See 'ClassifyImageTask' in the SNPE SDK for an example of this.