Multidimensional cv::Mat initialization and display

Question

Being a Matlab/Python guy and a novice in C++, I'm having major frustration moving to OpenCV in C++ for image processing purposes. I'm working with Kinect v2 so there is only one Windows example I found online which I'm modifying.

This example gives the depthMap as a cv::Mat and I've calculated surface normals on this depth image taken from a kinect v2. This surface normal image contains the i,j,k vector components (3 channels) per row,col element and I'm trying to visualize this surface normal image (3-D float matrix) as an RGB image. This is really easy to do in matlab since you just do an imshow(normMap) and it shows an RGB (ijk) image with the color specifying the orientation of the normal.

I'm trying to do a similar thing in C++. Since I'm using opencv, I decided to use the cv::Mat to store the ijk channels per pixel and I initialized the normal matrix (lets call it normMat) to a CV_32F matrix as follows:

int sizes[3] = { height, width, 3 };
cv::Mat normMat(3, sizes, CV_32F, cv::Scalar::all(0));

However, if I debug, the dims (normMat.rows and normMat.cols) are showing -1, so I don't know whether my initialization is bad or if I missed something, or whether it's normal.

I'm storing the surface normal components as:

normMat.at<float>(i, j, 0) = fVec3[0];
normMat.at<float>(i, j, 1) = fVec3[1];
normMat.at<float>(i, j, 2) = fVec3[2];

And they seem to be getting stored correctly as I've verified that in debug (using normMat.at<float>(i,j,k)).

Then I'm trying to display the image as:

normMat.convertTo(normColorMap, CV_8UC3, 255.0 / (4000), 255.0 / 500);
cv::imshow("Normals", normColorMap);

But the second line throws an exception:

OpenCV Error: Assertion failed (p[-1] 
<= 2) in cv::MatSize::operator.....

I also tried displaying normMat directly which throws the same exception. Which means that there's obviously something wrong with displaying a 3 channel image as a 3D matrix, or converting it to a 2D-3 Channel Mat. I also tried initializing normMat as

cv::Mat normMat(height, width, CV_32FC3, cv::Scalar::all(0));

But I'm having issues adding data to the 3 channels of a "2D 3 channel matrix" (CV_32FC3), if I may say it this way.

All I want to do is display this normMat as an RGB image. Any inputs,suggestions are highly appreciated. Thanks!