Error using CNN hyper parameter optimization for multi-target regression

Question

I am trying to solve a multi-target regression problem using CNN. To optimize hyper parameters, I have used Talos library, but I end up with an error:

Error when checking input: expected conv1d_1_input to have 3 dimensions, but got array with shape (280000, 70)

Description of Dataset: X has 70 features and Y has 32 targets Length of Time Series: 509760 time points

Shape of Dataset:

Train Data size: (400000, 70),
Train Target size: (400000, 32),
Test Data size: (109760, 70),
Test Target size: (109760, 32),

Code:

tf.keras.backend.clear_session()

def model(x_train, y_train, x_val, y_val, params):

    n_timesteps, n_features, n_outputs = x_train.shape[0],    x_train.shape[1], y_train.shape[1]

    model = Sequential()

    model.add(Conv1D(filters=params['conv1_filter'], kernel_size=(3), activation=params['activation'], input_shape=(n_timesteps,n_features)))
    model.add(Conv1D(filters=params['conv2_filter'], kernel_size=(3), activation=params['activation']))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Dropout(params['dropout']))
    model.add(Conv1D(filters=8, kernel_size=3, activation=params['activation']))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Dense(100, activation=params['activation']))
    model.add(Dense(n_outputs))

    model.add(Flatten())    
    model.compile(loss='mse',optimizer=params['optimizer'](),metrics=['acc', fmeasure_acc])


    out = model.fit(x_train, y_train, validation_data=[x_val, y_val], batch_size=params['batch_size'], epochs=params['epochs'],verbose=1,shuffle=True)

    return out, model

p = {'conv1_filter':[32],
     'conv2_filter':[32],
     'optimizer': [Nadam],
     'batch_size': [1000],
     'epochs': [5],
     'dropout': [0.5],
     'activation':[relu]}

# and run the experiment
t = ta.Scan(x=x_train,y=y_train,model=model,params=p,experiment_no='1')

The error is as follows:

    168         # input parameters section ends
    169 
--> 170         self._null = self.runtime()
    171 
    172     def runtime(self):

/anaconda3/lib/python3.7/site-packages/talos/scan/Scan.py in runtime(self)
    173 
    174         self = scan_prepare(self)
--> 175         self = scan_run(self)

/anaconda3/lib/python3.7/site-packages/talos/scan/scan_run.py in scan_run(self)
     16                      disable=self.disable_progress_bar)
     17     while len(self.param_log) != 0:
---> 18         self = scan_round(self)
     19         self.pbar.update(1)
     20     self.pbar.close()

/anaconda3/lib/python3.7/site-packages/talos/scan/scan_round.py in scan_round(self)
     30     # fit the model
     31     try:
---> 32         _hr_out, self.keras_model = ingest_model(self)
     33     except TypeError as err:
     34         if err.args[0] == "unsupported operand type(s) for +: 'int' and 'numpy.str_'":

/anaconda3/lib/python3.7/site-packages/talos/model/ingest_model.py in ingest_model(self)
      8                       self.x_val,
      9                       self.y_val,
---> 10                       self.round_params)

<ipython-input-9-b33b869928ed> in model(x_train, y_train, x_val, y_val, params)
     20 
     21 
---> 22     out = model.fit(x_train, y_train, validation_data=[x_val, y_val], batch_size=params['batch_size'], epochs=params['epochs'],verbose=1,shuffle=True)
     23 
     24 

/anaconda3/lib/python3.7/site-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
    950             sample_weight=sample_weight,
    951             class_weight=class_weight,
--> 952             batch_size=batch_size)
    953         # Prepare validation data.
    954         do_validation = False

/anaconda3/lib/python3.7/site-packages/keras/engine/training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, check_array_lengths, batch_size)
    749             feed_input_shapes,
    750             check_batch_axis=False,  # Don't enforce the batch size.
--> 751             exception_prefix='input')
    752 
    753         if y is not None:

/anaconda3/lib/python3.7/site-packages/keras/engine/training_utils.py in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
    126                         ': expected ' + names[i] + ' to have ' +
    127                         str(len(shape)) + ' dimensions, but got array '
--> 128                         'with shape ' + str(data_shape))
    129                 if not check_batch_axis:
    130                     data_shape = data_shape[1:]

ValueError: Error when checking input: expected conv1d_1_input to have 3 dimensions, but got array with shape (280000, 70)

Answer 1

I looked at it again, a few of things needed to be changed in your code. I tested it and it should work now:

First of all, the model itself had a couple of issues.

Namely, the input to your model should be (n_features, 1) instead of (n_timesteps,n_features). Keras internally thinks that you are ignoring the batch dimension when giving the input shape. So when what it will register as a shape is (batch_size, n_features, 1). The last dimension is required so that it is compatible with Conv1D.

This is how it should look:

model.add(Conv1D(filters=params['conv1_filter'], kernel_size=(3), activation=params['activation'], input_shape=(n_features, 1))) 

Secondly, the model shouldn't end with a Flatten() layer. Instead this layer should go before the Dense() layer (so that it sees a 2D input instead of a 3D one). After both changes the model should look like this:

model = Sequential()

model.add(Conv1D(filters=params['conv1_filter'], kernel_size=(3), activation=params['activation'], input_shape=(n_features,1))) 
model.add(Conv1D(filters=params['conv2_filter'], kernel_size=(3), activation=params['activation']))
model.add(MaxPooling1D(pool_size=2))
model.add(Dropout(params['dropout']))
model.add(Conv1D(filters=8, kernel_size=3, activation=params['activation']))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(100, activation=params['activation']))
model.add(Dense(n_outputs))

Because we changed the input_shape of the model, we'll need to change the shape of our data. What needs to be done is to change (400000, 70) to (400000, 70, 1) and (109760, 70) to (109760, 70, 1). This can be done with the following command:

x_train = np.expand_dims(x_train, axis=-1)
x_val = np.expand_dims(x_val, axis=-1)

I didn't test it on Talos, but the keras part works just fine.