What causes tensorflow keras Conv1D to only run the 1st epoch?

237 Views Asked by At

currently I am using tensorflow to create a neural network with a 1D convolutional layer and Dense layer to predict a single output value. The input array for the neural network is an array of 1500 samples; each sample is an array of 27x13 values. I started training in the same manner as I did without the 1D conv layer, but the training stopped during the first epoch without warning. I found that multiprocessing might be the cause and for that, I should turn multiprocessing off as discussed here: https://github.com/stellargraph/stellargraph/issues/1006

basically adding this to my keras model: use_multiprocessing=False

That did not change anything, after which I found that I should probably use a DataSet to bypass multiprocessing issues according to https://github.com/stellargraph/stellargraph/issues/1206

Replace tf.keras.Sequence objects with tf.data.Dataset #1206

after struggling with the difference between

tf.data.Dataset.from_tensors

and

tf.data.Dataset.from_tensor_slices 

I found the following code to start executing the model.fit block again. As you might have guessed, it still stops running after the first epoch:

main loop started
Epoch 1/5
Press any key to continue . . .

Can someone pinpoint the source of the halting of the program? This is my code:

import random
import numpy as np

from keras import backend as K

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from keras.models import load_model
from keras.callbacks import CSVLogger


EPOCHS = 5
BATCH_SIZE = 16

def tfdata_generator(x, y, is_training, batch_size=BATCH_SIZE):
  '''Construct a data generator using `tf.Dataset`. '''
  dataset = tf.data.Dataset.from_tensor_slices((x, y))

  if is_training:
    dataset = dataset.shuffle(1500)  # depends on sample size
  dataset = dataset.batch(BATCH_SIZE)
  dataset = dataset.repeat()
  dataset = dataset.prefetch(1)

  return dataset


def main():
    print("main loop started")
    X_train = np.random.randn(1500, 27, 13)
    Y_train = np.random.randn(1500, 1)    
    training_set = tfdata_generator(X_train, Y_train, is_training=True)
    data = np.random.randn(1500, 27, 13), Y_train
    training_set = tf.data.Dataset.from_tensors((X_train, Y_train))
    
    logstring = "C:\Documents\Conv1D"
    csv_logger = CSVLogger((logstring + ".csv"), append=True, separator=';')
    early_stopper = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=20, min_delta=0.00001)

    model = keras.Sequential()
    model.add(layers.Conv1D(
                filters=10,
                kernel_size=9,
                strides=3,
                padding="valid"))
    model.add(layers.Flatten())
    model.add(layers.Dense(70, activation='relu', name="layer2"))
    model.add(layers.Dense(1))

    optimizer =keras.optimizers.Adam(learning_rate=0.0001)
    model.compile(optimizer=optimizer, loss="mean_squared_error")    
    # WARNING:tensorflow:multiprocessing can interact badly with TensorFlow, causing nondeterministic deadlocks. For high performance data pipelines tf.data is recommended.

    model.fit(training_set, 
                epochs = EPOCHS, 
                batch_size=BATCH_SIZE,
                verbose = 2,
                #validation_split=0.2,
                use_multiprocessing=False);

    model.summary()
    modelstring = "C:\Documents\Conv1D_finishedmodel"
    model.save(modelstring, overwrite=True)
                
    model = load_model(modelstring)

main()
0

There are 0 best solutions below