I have been working with historical NVIDIA SMI outputs for a while. I haven't really seen a Fan Speed of more than 100% in value. But in a new dataset I am working with, I am seeing a few readings that are above 100%. How do I interpret this?
From the official documentation:
The fan speed value is the percent of maximum speed that the device's fan is currently intended to run at. It ranges from 0 to 100%. Note: The reported speed is the intended fan speed. If the fan is physically blocked and unable to spin, this output will not match the actual fan speed. Many parts do not report fan speeds because they rely on cooling via fans in the surrounding enclosure. For all discrete products with dedicated fans.
Despite that, I am seeing the following in my readings that were collected sometime between Sept 2021 and Oct 2021.
Mon Sep 27 04:10:01 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 On | 00000000:04:00.0 Off | N/A |
| 27% 22C P8 7W / 180W | 0MiB / 8119MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 1080 On | 00000000:05:00.0 Off | N/A |
| 27% 23C P8 7W / 180W | 0MiB / 8119MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 1080 On | 00000000:08:00.0 Off | N/A |
| 49% 73C P2 101W / 180W | 1053MiB / 8119MiB | 89% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 1080 On | 00000000:09:00.0 Off | N/A |
| 55% 81C P2 62W / 180W | 1063MiB / 8119MiB | 84% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 GeForce GTX 1080 On | 00000000:84:00.0 Off | N/A |
| 27% 25C P8 6W / 180W | 0MiB / 8119MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 5 GeForce GTX 1080 On | 00000000:85:00.0 Off | N/A |
|635% 82C P2 158W / 180W | 347MiB / 8119MiB | 84% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 6 GeForce GTX 1080 On | 00000000:88:00.0 Off | N/A |
| 27% 19C P8 7W / 180W | 0MiB / 8119MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 7 GeForce GTX 1080 On | 00000000:89:00.0 Off | N/A |
| 47% 70C P2 103W / 180W | 293MiB / 8119MiB | 76% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
Sat Oct 9 09:00:02 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 On | 00000000:04:00.0 Off | N/A |
| 40% 63C P2 130W / 180W | 237MiB / 8119MiB | 76% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 1080 On | 00000000:05:00.0 Off | N/A |
| 47% 82C P2 151W / 180W | 253MiB / 8119MiB | 79% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 1080 On | 00000000:08:00.0 Off | N/A |
| 49% 75C P2 120W / 180W | 241MiB / 8119MiB | 76% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 1080 On | 00000000:09:00.0 Off | N/A |
| 54% 82C P2 146W / 180W | 253MiB / 8119MiB | 78% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 GeForce GTX 1080 On | 00000000:84:00.0 Off | N/A |
| 49% 72C P2 108W / 180W | 259MiB / 8119MiB | 77% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 5 GeForce GTX 1080 On | 00000000:85:00.0 Off | N/A |
|541% 82C P2 155W / 180W | 253MiB / 8119MiB | 79% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 6 GeForce GTX 1080 On | 00000000:88:00.0 Off | N/A |
| 48% 70C P2 161W / 180W | 965MiB / 8119MiB | 86% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 7 GeForce GTX 1080 On | 00000000:89:00.0 Off | N/A |
| 47% 72C P2 118W / 180W | 289MiB / 8119MiB | 78% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
Wed Sep 22 20:20:01 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.80 Driver Version: 460.80 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 On | 00000000:04:00.0 Off | N/A |
| 48% 73C P2 167W / 180W | 955MiB / 8119MiB | 87% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 1080 On | 00000000:05:00.0 Off | N/A |
| 53% 81C P2 166W / 180W | 959MiB / 8119MiB | 89% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 1080 On | 00000000:08:00.0 Off | N/A |
| 52% 78C P2 156W / 180W | 955MiB / 8119MiB | 88% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 1080 On | 00000000:09:00.0 Off | N/A |
| 54% 82C P2 160W / 180W | 955MiB / 8119MiB | 90% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 GeForce GTX 1080 On | 00000000:84:00.0 Off | N/A |
| 43% 69C P2 154W / 180W | 263MiB / 8119MiB | 82% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 5 GeForce GTX 1080 On | 00000000:85:00.0 Off | N/A |
|739% 76C P2 165W / 180W | 263MiB / 8119MiB | 85% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 6 GeForce GTX 1080 On | 00000000:88:00.0 Off | N/A |
| 42% 67C P2 172W / 180W | 263MiB / 8119MiB | 84% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 7 GeForce GTX 1080 On | 00000000:89:00.0 Off | N/A |
| 46% 73C P2 166W / 180W | 263MiB / 8119MiB | 87% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
Also, this usually occurs on GPU index 5 of a particular host, what can one hypothesize from that?