HEVC decoding with VA-API is not recognizing my device

Question

I'm using libva to decode HEVC binary file. The following to code to decode a single HEVC encoded frame.

int main() {
   int drm_fd = open("/dev/dri/card0", O_RDWR); // OK
   VADisplay display = vaGetDisplayDRM(drm_fd); // OK
   int major, minor;
   vaInitialize(display, &major, &minor);       // runtime-error: Error code 0xffffffff: 
   // more code 
   return 0; 
}

The error thrown is not descriptive unfortunately. Error code 0xffffffff: unknown libva error

I get same result if I use /dev/dri/render128 as well.

vainfo looks healthy

libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_20
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.20 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 23.2.1-1+build1 for AMD Radeon Pro WX 3200 Series (polaris12, LLVM 16.0.6, DRM 3.54, Debian-12-amd64)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

I can see that ffmpeg + vaapi works fine too, this command runs fine

$ ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/card0  -f hevc -i inputencodedsingleframe out.png

Some additional info on my setup

$ drm_info -j | jq 'with_entries(.value |= .driver.desc)'
{
  "/dev/dri/card0": "AMD GPU"
}

$ lspci | grep VGA
61:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] (rev 10)

I am starting to believe that VA-api doesn't work with AMD graphic cards but the fact that vainfo and ffmpeg works fine, tell me that might be something else. This code works fine in another linux machine so I don't know what can be different. Any pointers will be appreciated.