How to convert an OpenNI VideoFrame to an OpenCV Mat data structure?

Question

I am looking for general descriptions / specifications on the format of an openni VideoFrame object which I am trying to convert to opencv Mat.

Is there some offical website/documentation available? I have found the following on ros.org which describes the VideoFrameRef and OniFrame class. But first this seems to be related to libfreenect and secondly I dont find any specific information about the structure of the frame (like the dimensions, channels, etc.).

So any links or descriptions would be greatly appreciated!

The code below is a snippet I am experimenting with. The conversion part within the while loop only produces a black video stream.

import sys
import numpy as np
import cv2
from openni import openni2
from openni import _openni2 as c_api

openni2.initialize("./Redist")

device = openni2.Device.open_any()

ir_stream = device.create_ir_stream()
ir_stream.set_video_mode(c_api.OniVideoMode(pixelFormat = c_api.OniPixelFormat.ONI_PIXEL_FORMAT_GRAY16, resolutionX = 320, resolutionY = 240, fps = 30))
ir_stream.start()

while(True):
    frame = ir_stream.read_frame()
    frame_data = frame.get_buffer_as_uint16()
    img = np.frombuffer(frame_data, dtype=np.uint16)
    img.shape = (1, 240, 320)
    img = np.concatenate((img, img, img), axis=0)
    img = np.swapaxes(img, 0, 2)
    img = np.swapaxes(img, 0, 1)
    cv2.imshow("image", img)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

ir_stream.stop()
openni2.unload()

Answer 1

I would like to reply to my own question as I am sure others could profit from this working example.

cv2.imshow() understands uint8 (0..255), uint32(0..2**32-1), and float (0..1) so we have to cast the array using img.astype(np.float) / 1024

import numpy as np
import cv2
from openni import openni2
from openni import _openni2 as c_api

openni2.initialize("./Redist")
device = openni2.Device.open_any()

ir_stream = device.create_ir_stream()
ir_stream.set_video_mode(c_api.OniVideoMode(pixelFormat = c_api.OniPixelFormat.ONI_PIXEL_FORMAT_GRAY16, resolutionX = 320, resolutionY = 240, fps = 30))
ir_stream.start()

while(True):
    frame = ir_stream.read_frame()

    # returns a 1-dim c_ushort_array of len: 76'800 = 320x240 each pixel having an uint16 value 0..65535
    frame_data = frame.get_buffer_as_uint16()

    # converting to numpy.ndarray which is still 1-dimension only
    img = np.frombuffer(frame_data, dtype=np.uint16)

    # convert to 3-dimensional array
    img.shape = (240, 320)

    # normalize values 
    img = img.astype(np.float) / 1024

    # Display image
    cv2.imshow("image", img)

    # Wait for input
    if cv2.waitKey(1) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

ir_stream.stop()
openni2.unload()

Related stackoverflow posts:

• Getting data from ctypes array into numpy
• OpenNI and OpenCV: cv2.imshow() crashes with error: (-215:Assertion failed) dst.data == (uchar*)dst_ptr in function 'cvShowImage'