im using YOLOv4 custom function from the AI guys's repo. Everything works flawlessly, including convert weights file to pb file, run detection on both image and videos, then this error shows up on both pc and colab:
- When i use only CPU, detection runs fine on both image and video data
- When i use GPU, detections only runs on image. When run on videos, it still runs, but doesnt detect anything. I assume that the detection only runs on the first frame, but not on the rest ones.
- Here is the console log when run video detect on CPU :
2021-05-01 11:19:05.173287: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-05-01 11:19:07.268691: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-05-01 11:19:07.269701: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2021-05-01 11:19:07.279142: E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected 2021-05-01 11:19:07.279180: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (76ac529fe3d6): /proc/driver/nvidia/version does not exist 2021-05-01 11:19:07.325497: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set True: False 2021-05-01 11:19:07.328773: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set Current threshold: 0.5 2021-05-01 11:19:23.595602: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-05-01 11:19:23.658932: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2299995000 Hz FPS: 0.77 Object found: Noentry, Confidence: 0.69, BBox Coords (xmin, ymin, xmax, ymax): 380.0, 389.0, 392.0, 397.0 FPS: 2.60 Object found: Noentry, Confidence: 0.68, BBox Coords (xmin, ymin, xmax, ymax): 380.0, 389.0, 392.0, 397.0 FPS: 2.38 Object found: Noentry, Confidence: 0.69, BBox Coords (xmin, ymin, xmax, ymax): 380.0, 389.0, 392.0, 396.0 FPS: 2.51 Object found: Noentry, Confidence: 0.89, BBox Coords (xmin, ymin, xmax, ymax): 380.0, 389.0, 391.0, 397.0 FPS: 2.48 Object found: Noentry, Confidence: 0.90, BBox Coords (xmin, ymin, xmax, ymax): 380.0, 390.0, 391.0, 397.0 FPS: 2.52 Object found: Noentry, Confidence: 0.90, BBox Coords (xmin, ymin, xmax, ymax): 380.0, 390.0, 391.0, 397.0 FPS: 2.64 Object found: Noentry, Confidence: 0.87, BBox Coords (xmin, ymin, xmax, ymax): 379.0, 391.0, 390.0, 398.0 FPS: 2.59 Object found: Noentry, Confidence: 0.76, BBox Coords (xmin, ymin, xmax, ymax): 378.0, 392.0, 390.0, 399.0 FPS: 2.64 Object found: Noentry, Confidence: 0.75, BBox Coords (xmin, ymin, xmax, ymax): 377.0, 392.0, 390.0, 400.0 FPS: 2.61 Object found: Noentry, Confidence: 0.71, BBox Coords (xmin, ymin, xmax, ymax): 377.0, 393.0, 389.0, 401.0 FPS: 2.58 FPS: 2.66 FPS: 2.52 FPS: 2.50 FPS: 2.57 FPS: 2.59 FPS: 2.60 FPS: 2.75 FPS: 2.57 FPS: 2.77
- Here is the console log when run video detection on GPU ( im sorry if log is a little bit too long )
2021-05-01 10:32:33.098862: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-05-01 10:32:34.935938: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-05-01 10:32:34.936785: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2021-05-01 10:32:34.953709: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:34.954371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0 coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s 2021-05-01 10:32:34.954406: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-05-01 10:32:34.957049: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-05-01 10:32:34.957129: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2021-05-01 10:32:34.958917: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-05-01 10:32:34.959305: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-05-01 10:32:34.961111: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-05-01 10:32:34.961867: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2021-05-01 10:32:34.962058: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-05-01 10:32:34.962151: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:34.962729: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:34.963310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2021-05-01 10:32:35.006935: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-05-01 10:32:35.007063: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.007689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0 coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s 2021-05-01 10:32:35.007719: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-05-01 10:32:35.007756: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-05-01 10:32:35.007781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2021-05-01 10:32:35.007803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-05-01 10:32:35.007839: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-05-01 10:32:35.007859: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-05-01 10:32:35.007881: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2021-05-01 10:32:35.007903: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-05-01 10:32:35.007991: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.008575: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.009095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2021-05-01 10:32:35.009140: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-05-01 10:32:35.503454: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-05-01 10:32:35.503505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 2021-05-01 10:32:35.503515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N 2021-05-01 10:32:35.503690: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.504369: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.504897: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.505452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14975 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) True: False 2021-05-01 10:32:35.506749: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-05-01 10:32:35.506845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.507388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0 coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s 2021-05-01 10:32:35.507418: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-05-01 10:32:35.507459: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-05-01 10:32:35.507472: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2021-05-01 10:32:35.507485: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-05-01 10:32:35.507497: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-05-01 10:32:35.507512: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-05-01 10:32:35.507524: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2021-05-01 10:32:35.507536: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-05-01 10:32:35.507582: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.508113: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.508604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2021-05-01 10:32:35.508630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-05-01 10:32:35.508640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 2021-05-01 10:32:35.508647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N 2021-05-01 10:32:35.508710: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.509240: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-05-01 10:32:35.509738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14975 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) Current threshold: 0.5 2021-05-01 10:32:50.238825: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-05-01 10:32:50.290412: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2199995000 Hz 2021-05-01 10:32:50.932916: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-05-01 10:32:51.960900: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-05-01 10:32:52.189146: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 FPS: 0.41 FPS: 23.86 FPS: 23.61 FPS: 24.18 FPS: 23.84 FPS: 23.82 FPS: 24.36 FPS: 23.82 FPS: 24.10 FPS: 24.39 FPS: 24.17 FPS: 24.09 FPS: 24.54 FPS: 23.93 FPS: 23.35 FPS: 24.02 FPS: 24.11 FPS: 24.48 FPS: 24.10 FPS: 24.29 FPS: 24.46 FPS: 24.48 FPS: 23.62 FPS: 23.79 FPS: 23.76 FPS: 24.01 FPS: 24.43 FPS: 23.52 FPS: 24.46 FPS: 24.13 FPS: 24.11 FPS: 23.79 FPS: 24.62 FPS: 24.32 FPS: 24.06 FPS: 24.51 FPS: 24.46 FPS: 24.40 FPS: 24.48 FPS: 24.62 Video end
At first i assume that my GPU is not strong enough ( GTX 1050 mobile), but it happens even when running on colab pro with Tesla V100 I appreciate any helps.
I faced the same problem and this is how I solved it. The problem is caused by TensorFlow's version. Downgrade the version to 2.3.0.