Emgu CV, unable to add Image to Matrix for KNN training

Question

I'm really stumped on this one, any assistance would be greatly appreciated. I'm attempting to translate the following:

https://github.com/MicrocontrollersAndMore/OpenCV_KNN_Character_Recognition_Machine_Learning/blob/master/generate_data.cpp

which is written in C++ OpenCV, to Emgu CV, preferably using VB, but C# would be fine as well. I'm using Emgu CV 2.4.10 currently (holding off on moving to >= 3.X until Emgu 3.X is passed release candidate stage).

Where I'm having trouble is towards the end, where the training images have to be added to a OpenCV Matrix before this Matrix is passed into the KNN call to train. Here is what I have so far in the button click event to open the file with the training numbers:

Dim imgTrainingNumbers As Image(Of Bgr, Byte)
imgTrainingNumbers = New Image(Of Bgr, Byte)(ofdOpenFile.FileName)             'open image
'some error checking for verifying the image opened omitted here, its in the actual program


Dim imgGrayscale As Image(Of Gray, Byte)
Dim imgBlurred As Image(Of Gray, Byte)
Dim imgThresh As Image(Of Gray, Byte) = Nothing
Dim imgThreshCopy As Image(Of Gray, Byte)
Dim imgContours As Image(Of Gray, Byte)

Dim contours As Contour(Of Point)

Dim mtxClassificationInts As Matrix(Of Single) = New Matrix(Of Single)(NUMBER_OF_TRAINING_SAMPLES, 1)
Dim mtxTrainingImages As Matrix(Of Single) = New Matrix(Of Single)(RESIZED_IMAGE_WIDTH * RESIZED_IMAGE_HEIGHT * NUMBER_OF_TRAINING_SAMPLES, 1)

Dim intValidChars As New List(Of Integer)(New Integer() { 48, 49, 50, 51, 52, 53, 54, 55, 56, 57 })

imgGrayscale = imgTrainingNumbers.Convert(Of Gray, Byte)()             'convert to grayscale
imgBlurred = imgGrayscale.SmoothGaussian(5)

CvInvoke.cvShowImage("imgBlurred", imgBlurred)

imgThresh = imgBlurred.ThresholdAdaptive(New Gray(255), ADAPTIVE_THRESHOLD_TYPE.CV_ADAPTIVE_THRESH_GAUSSIAN_C, THRESH.CV_THRESH_BINARY_INV, 11, New Gray(2))

imgThreshCopy = imgThresh.Clone()

contours = imgThreshCopy.FindContours(CHAIN_APPROX_METHOD.CV_CHAIN_APPROX_SIMPLE, RETR_TYPE.CV_RETR_EXTERNAL)

imgContours = New Image(Of Gray, Byte)(imgThresh.Size())

CvInvoke.cvDrawContours(imgContours, contours, New MCvScalar(255), New MCvScalar(255), 100, 1, LINE_TYPE.CV_AA, New Point(0, 0))

CvInvoke.cvShowImage("imgThresh", imgThresh)
CvInvoke.cvShowImage("imgContours", imgContours)

While(Not contours Is Nothing)
    If (contours.Area > MIN_CONTOUR_AREA) Then
        Dim rect As Rectangle = contours.BoundingRectangle()            'get the bounding rect
        imgTrainingNumbers.Draw(rect, New Bgr(Color.Red), 2)            'draw red rect around the current char
        Dim imgROI As Image(Of Gray, Byte) = imgThresh.Copy(rect)

        Dim imgROIResized As Image(Of Gray, Byte) = imgROI.Resize(RESIZED_IMAGE_WIDTH, RESIZED_IMAGE_HEIGHT, INTER.CV_INTER_LINEAR)


        CvInvoke.cvShowImage("imgROI", imgROI)
        CvInvoke.cvShowImage("imgROIResized", imgROIResized)
        CvInvoke.cvShowImage("imgTrainingNumbers", imgTrainingNumbers)

        Dim intChar As Integer = CvInvoke.cvWaitKey(0)

        If (intChar = 27) Then
            'add code to exit program here if Esc is pressed
        ElseIf (intValidChars.Contains(intChar)) Then
            mtxClassificationInts.Add(intChar)      'append classification char to matrix of integers (we will convert later before writing to file)

            'now add the training image (some conversion is necessary first) . . .

            Dim mtxTemp As Matrix(Of Single) = New Matrix(Of Single)(imgROIResized.Size())
            Dim mtxTempReshaped As Matrix(Of Single) = New Matrix(Of Single)(imgROIResized.Size())

            CvInvoke.cvConvert(imgROIResized, mtxTemp)

            mtxTempReshaped = mtxTemp.Reshape(1, 1)

            Try
                mtxTrainingImages.Add(mtxTempReshaped)
            Catch ex As Exception
                txtInfo.Text = txtInfo.Text + ex.Message + vbCrLf
            End Try

        End If

    End If
    contours = contours.HNext
End While

Me.Text = "training complete !!"

'write mtxClassificationInts to file here
'write mtxTrainingImages to file here

'separate write and read into two separate programs when all this is working

'read mtxClassificationInts to file
'read mtxTrainingImages to file    

Dim kNearest As KNearest = New KNearest()                   'instantiate KNN object

kNearest.Train(mtxTrainingImages, mtxClassificationInts, Nothing, False, 1,False)       'call to train

'rest of program here when training is successful

In the Try . . . Catch block, on the line:

mtxTrainingImages.Add(mtxTempReshaped)

I'm getting the following error:

OpenCV: The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array'

I've tried about every type of format change I can find but can't seem to get past an error on this line.

I should probably mention a few other things:

-The KNN call to train only accepts Matrix of type Single (float if using #C, same thing), so it has to be in this format

-I got the example:

http://www.emgu.com/wiki/index.php/K_Nearest_Neighbors_in_CSharp

to work in both C# and VB, but I'm not sure how to apply this to using actual images rather than made up random numbers

-Yes, I'm aware Emgu has Tesseract built in for character recognition, but I plan on moving onto other machine learning in Emgu and would like to get this working first as a relatively easy example

Any help would be great.

Answer 1

I believe the error message is telling you that mtxTrainingImages and mtxTempReshaped are different sizes and/or have a different number of channels. If these two are created the same you won't have this problem.