How to extract motion vectors from h264 without a full decode

Question

I'm want to extract the motion vector information from the encoded stream without performing a full decode of the frame. I'm using ffmpeg library and aware of extracting the information using the av_frame_get_side_data. Unfortunately, it does a full decode of the frame before providing the AV_FRAME_DATA_MOTION_VECTORS. My hunch is it should be possible to parse the h264 to extract out the motion vectors only, ignoring all the other data. Any direction/pointers will be very helpful!