How do I know if koboldcpp is using my GPU?

Question

I have compiled koboldcpp from source on Ubuntu 18.04 using:

git clone https://github.com/LostRuins/koboldcpp
cd koboldcpp
make -j10 koboldcpp_cublas LLAMA_OPENBLAS=1 LLAMA_CUBLAST=1

But when it loads it does not use my GPU (I checked using nvidia-smi and it's at 0%). It uses CPU only.

koboldcpp3  | Identified as LLAMA model: (ver 1)
koboldcpp3  | System Info: AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |

Does koboldcpp log explicitly whether it is using the GPU, i.e. printf("I am using the GPU\n"); vs printf("I am using the CPU\n"); so I can learn it straight from the horse's mouth instead of relying on external tools such as nvidia-smi?

Should I look for BLAS = 1 in the System Info log?

Answer 1

Hm, according to https://github.com/LostRuins/koboldcpp/discussions/21:

This tool currently only focuses on CPU inference, you may want to check out this repo instead for GPU inference for llama: https://github.com/0cc4m/KoboldAI

I guess I have been wasting my time trying to get the GPU to work with koboldcpp