DeBayering a Canon CR2 image with OpenCV

Question

What I'm trying to do is read a .cr2 RAW file taken on my Canon Rebel T5 1200D. I'm able to load and display the image using LibRaw and OpenCV, but the image appears to be slightly too bright and a bit more yellow than it does when I open the file in Windows. The images themselves are at the bottom of the post. A few similar questions have been asked:

Using LibRaw to correctly decode CR2 image?

Convert 12-bit Bayer image to 8-bit RGB using OpenCV

I'll first post all of my code, then get into the details of the problem.

// Open the CR2 file with LibRaw, unpack, and create image
LibRaw lrProc;
assert( LIBRAW_SUCCESS == lrProc.open_file( "001.cr2" ) );
assert( LIBRAW_SUCCESS == lrProc.unpack() );
assert( LIBRAW_SUCCESS == lrProc.raw2image() );

// Get image dimensions
int width = lrProc.imgdata.sizes.iwidth;
int height = lrProc.imgdata.sizes.iheight;

// Create a buffer of ushorts containing the pixel values of the "BG Bayered" image
std::vector<ushort> vBayerData;
for ( int y = 0; y < height; y++ )
{
    for ( int x = 0; x < width; x++ )
    {
        // Get pixel idx
        int idx = y * width + x;

        // Each pixel is an array of 4 shorts rgbg
        ushort * uRGBG = lrProc.imgdata.image[idx];

        // Even rows are RGRGRG..., odd are GBGBGB...
        // For even rows, get either red or green, store in vec
        if ( y % 2 == 0 )
        {
            bool red = x % 2 == 0;
            vBayerData.push_back( uRGBG[red ? 0 : 1] );
        }
        // For odd rows, get either blue or green
        else
        {
            bool green = x % 2 == 0;
            vBayerData.push_back( uRGBG[green ? 3 : 2] );
        }
    }
}

// Get rid of libraw image, construct openCV mat
lrProc.recycle();
cv::Mat imgBayer( height, width, CV_16UC1, vBayerData.data() );

// Debayer image, get output
cv::Mat imgDeBayer;
cv::cvtColor( imgBayer, imgDeBayer, CV_BayerBG2RGB );

// The pixel color values were 12 bit, but our data is 16 bit
// transform the range [0, 4095] to [0:65535] (multiply by 16)
imgDeBayer *= 16;

// Display image
cv::namedWindow( "CR2 File", CV_WINDOW_FREERATIO );
cv::imshow( "CR2 File", imgDeBayer );
cv::waitKey();

I started by using LibRaw to open and unpack the raw image file. This example on LibRaw's website http://www.libraw.org/docs/API-overview-eng.html

indicates that I'll be left with an "image" where each pixel is actual 4 ushort values [Red, Green1, Blue, Green2] representing the bayered color at that pixel. When I print out the values (using the following code)

// Print the first 4 values of the first 4 rows
for ( int y = 0; y < 4; y++ )
{
    for ( int x = 0; x < 4; x++ )
    {
        int idx = y * width + x;
        ushort * uRGBG = lrProc.imgdata.image[idx];
        printf( "[%04d, %04d, %04d, %04d]  ", uRGBG[0], uRGBG[1], uRGBG[2], uRGBG[3] );
    }
    printf( "\n" );
}

I get the following results:

[2253, 0000, 0000, 0000]  [0000, 2166, 0000, 0000]  [2183, 0000, 0000, 0000]  [0000, 2195, 0000, 0000]
[0000, 0000, 0000, 2207]  [0000, 0000, 2175, 0000]  [0000, 0000, 0000, 2099]  [0000, 0000, 2122, 0000]
[2246, 0000, 0000, 0000]  [0000, 2240, 0000, 0000]  [2287, 0000, 0000, 0000]  [0000, 2182, 0000, 0000]
[0000, 0000, 0000, 2251]  [0000, 0000, 2103, 0000]  [0000, 0000, 0000, 2195]  [0000, 0000, 2155, 0000]

So on even rows the Red and Green1 pixels have alternatively nonzero values, and on odd rows the Blue and Green2 pixels have alternatively nozero values. The values seem to be 12 bit, although I'm not 100% on that.

Looking at the description of OpenCV's cvtColor function

http://docs.opencv.org/2.4/modules/imgproc/doc/miscellaneous_transformations.html#cvtcolor

As well as this person's description of some of the cr2 color formats:

http://lclevy.free.fr/cr2/#interpol

Indicates to me that I've got a Bayered image of the "BG" variety (going by the terminology used in the OpenCV docs.)

So, in order to pass that image to cvtColor I've got to take each pixel value from the unpacked raw image and create a contiguous image of unsigned shorts. // Get pixel idx int idx = y * width + x;

// Each pixel is an array of 4 shorts rgbg
ushort * uRGBG = lrProc.imgdata.image[idx];

// For even rows, get either red or green, store in vec
if ( y % 2 == 0 )
{
    bool red = x % 2 == 0;
    vBayerData.push_back( uRGBG[red ? 0 : 1] );
}
// For odd rows, get either blue or green
else
{
    bool green = x % 2 == 0;
    vBayerData.push_back( uRGBG[green ? 3 : 2] );
}

Once I've got a contiguous buffer of shorts (vBayerData), I construct an opencv mat and debayer it using CV_BayerBG2RGB.

However, looking at the pixel values I printed out tells me that the values are probably 12 bit (although I'm not entirely sure.) So I've got to go from a 12 bit range to a 16 bit range, which I believe amounts to multiplying the values by 16.

// Construct bayer image from data
cv::Mat imgBayer( height, width, CV_16UC1, vBayerData.data() );

// Debayer image, get output
cv::Mat imgDeBayer;
cv::cvtColor( imgBayer, imgDeBayer, CV_BayerBG2RGB );

// The pixel color values were 12 bit, but our data is 16 bit
// transform the range [0, 4095] to [0:65535] (multiply by 16)
imgDeBayer *= 16;

After all that, here is the image I'm left with:

However, here is what my eyes see (sorry about the scaling, color is what matters):

I feel like I'm so close here, but the image I'm getting is a little too bright and a little off color. The settings on my camera are to export RAW images using sRGB format, so I was sure that applying some sort of gamma correction to the image intensity would help, but I can't seem to get the correct image out.

I've tried different Bayer matrices (i.e RG), different bit depth conversions, rearranging when I do the bit depth conversion (before and after DeBayering), but none seem to make the image appear correctly.

Does anyone recognize this kind of discoloration? And if so, could they assist me in finding the error in my code?

Thank you for reading,

John

P.S I know there's a lot of text here, I wasn't sure what the best format for a post like this was. I just wanted to provide all the information I'm working with.

EDIT: As Mark Ransom suggested below, I think this is due to an incorrect white balance. I was pointed in the direction of the digital camera color pipeline, and there is much more to decoding an image than just the debayer step.

From the wikipedia article:

Typical components include image sensor corrections (including "debaying" or applying a Bayer filter), noise reduction, image scaling, gamma correction, image enhancement, colorspace conversion (between formats such as RGB, YUV or YCbCr), chroma subsampling, framerate conversion, image compression/video compression (such as JPEG), and computer data storage/data transmission.