ValueError: Shapes (None, 8) and (None, 1) are incompatible when running model.fit() from Keras

21 Views Asked by At

Here's the error: error

"/opt/conda/lib/python3.10/site-packages/keras/utils/metrics_utils.py", line 674, in update_confusion_matrix_variables
        y_pred.shape.assert_is_compatible_with(y_true.shape)

    ValueError: Shapes (None, 8) and (None, 1) are incompatible

Here's the code that produced it:

#results of gridSearch...WHICH TOOK ALL NIGHT OMG
activation = 'relu'
batch_size= 10
epochs=15
learning_rate=.001
rho = .95

model = create_model(learning_rate, rho, activation, batch_size)

model.summary()

print(FEATURES_train.shape)
print(LABELS_train.shape)

# validation_data=(FEATURES_validate, LABELS_validate)
history = model.fit(FEATURES_train, LABELS_train, epochs=epochs)

Here's the model.summary and shape of FEATURES_train and LABELS_train right before entering CNN:

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 96, 96, 6)         456       
                                                                 
 average_pooling2d (AverageP  (None, 48, 48, 6)        0         
 ooling2D)                                                       
                                                                 
 conv2d_1 (Conv2D)           (None, 44, 44, 16)        2416      
                                                                 
 average_pooling2d_1 (Averag  (None, 22, 22, 16)       0         
 ePooling2D)                                                     
                                                                 
 flatten (Flatten)           (None, 7744)              0         
                                                                 
 dense (Dense)               (None, 400)               3098000   
                                                                 
 dense_1 (Dense)             (None, 180)               72180     
                                                                 
 dense_2 (Dense)             (None, 8)                 1448      
                                                                 
=================================================================
Total params: 3,174,500
Trainable params: 3,174,500
Non-trainable params: 0
_________________________________________________________________
(1362, 100, 100, 3)
(1362,)

Here's where the model is defined. It's a CNN based on LeNet-5 architecture:

# Define a function to create the model
def create_model(learning_rate, rho, activation, batch_size):
    model = Sequential([
        Conv2D(6, (5, 5), activation=activation, input_shape=(IMG_SIZE, IMG_SIZE, 3)),
        AveragePooling2D(pool_size=(2, 2), strides=2),
        Conv2D(16, (5, 5), activation=activation),
        AveragePooling2D(pool_size=(2, 2), strides=2),
        Flatten(),
        Dense(400, activation=activation),
        Dense(180, activation=activation),
        Dense(len(CATEGORIES), activation='softmax')
    ])

    
    rmsprop = RMSprop(learning_rate=learning_rate, rho=rho)

    model.compile(optimizer=rmsprop, loss='sparse_categorical_crossentropy', metrics=['accuracy', Precision(), Recall(), AUC()])

    return model

Here's where I load the data:


# Categories which CNN will classify by -----------------------------------------------
CATEGORIES = ["0", "1", "2", "5", "10", "20", "50", "100"]

# The size of the images that your neural network will use
IMG_SIZE = 100

#Arrays to hold features and labels
FEATURES = []
LABELS = []

for c in CATEGORIES:  # access all folders
    dataPath = os.path.join(DATADIR, c)  # create a path for each folder
    classNum = CATEGORIES.index(c)
    img_files = (img for img in os.listdir(dataPath) if img.endswith(".jpg"))
    for img in img_files: # access all images in each folder
        #preprocessing
        #read image, in color
        img = cv.imread(os.path.join(dataPath, img), cv.IMREAD_COLOR)
        #Ensure every image is the same size
        img = cv.resize(img,(IMG_SIZE, IMG_SIZE) )
        #normalize
        img = img.astype(np.float32)/255
        #add image & label to arrays
        FEATURES.append(img)
        LABELS.append(classNum)
        
        
#turn pictures and labels lists into numpy array        
X = np.array(FEATURES)
y = np.array(LABELS)

#split data into train(70%), test(15%), and validation(15%) datasets Shuffled and stratified to maintain same proportions of upsampled classes
FEATURES_train, FEATURES_rest, LABELS_train, LABELS_rest = train_test_split(X, y, train_size = .7, shuffle=True)  

It looks like maybe there's some problem when adding values to confusion matrix but why is that? Am I on the right track?

How do I solve this?

I verified the shape of FEATURES_train and LABELS_train. They look like they're correct.

I verified that I was using sparse_categorical_crossentropy. No problem there.

I had included a validation dataset as a argument in model.fit, but removed it to see if that was causing the problem. The problem still persisted. (Maybe I should add the validation set back in.)

I just included validation_split=.2 and the issue still persists.

0

There are 0 best solutions below