I've been trying to implement a GA to optimize the parameters of my ANN. I'm new to both of these libraries and I've used this as help to implement it https://blog.paperspace.com/train-keras-models-using-genetic-algorithm-with-pygad/. I'm using: pygad version 3.0.1 for the GA, Tensorflow keras version 2.12.0 for the ANN model, I have numpy version 1.23.5, my python3 version is 3.10.6.
The error i encounter is the following (however it should be noted that sometimes the error does not occur during a run, at other times it has occured hundreds of times during a run, and i can not see any patern in when it occurs. This is despite me using the same input data all the time):
Traceback (most recent call last):
File "/home/carl/dev/emg_processing/src/./main.py", line 133, in <module>
main(sys.argv[1:])
File "/home/carl/dev/emg_processing/src/./main.py", line 120, in main
classifier = ann(segment_arr, label_arr, args.k, args.dr, input_dim, args.l, args.sf, args.i, args.af, args.n, args.bs, args.nc, args.ns, args.ng, args.npm)
File "/home/carl/dev/emg_processing/src/classifier/classifier.py", line 150, in ann
best_solution_weights = ga(num_solutions, num_generations, num_parents_mating)
File "/home/carl/dev/emg_processing/src/optimizer/optimizer.py", line 78, in ga
ga_instance.run()
File "/home/carl/.local/lib/python3.10/site-packages/pygad/pygad.py", line 1875, in run
best_solution, best_solution_fitness, best_match_idx = self.best_solution(pop_fitness=self.last_generation_fitness)
File "/home/carl/.local/lib/python3.10/site-packages/pygad/pygad.py", line 1963, in best_solution
best_match_idx = numpy.where(pop_fitness == numpy.max(pop_fitness))[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0
However i've been able to find that this issue is the result of the fitness function returning NaN, which in turn is the result of the categorical cross entropy returning NaN. I've checked the predictions and the labels provided to the cross categorical entropy and i can't find anything that would answer why this would result in NaN, to me it looks reasonable. So im wondering if anyone has encountered this aswell and if anyone knows how to fix it.
The following is the code for the GA, im using a simple 2 layer ANN. My data has 11 possible classes. All 'config.' are imported global variables since fitness function only accepts three specific arguments.
import pygad.kerasga
import numpy
import tensorflow.keras
import config
#==============================================================================
# Fitness function for calculating the fitness of a solution
def fitness_func(ga_instance, solution, sol_idx):
# Obtain the parameters from one solution
model_weights_matrix = pygad.kerasga.model_weights_as_matrix(model=config.model, weights_vector=solution)
# Use the solutions parameters to set the models parameters
config.model.set_weights(weights=model_weights_matrix)
# The model predicts on all the data
predictions = config.model.predict(config.ga_data, verbose = 0)
# We check how well it predicted and base it's fitness on how good it predicted
cce = tensorflow.keras.losses.CategoricalCrossentropy()
# Add the small value so we don't divide by 0
solution_fitness = 1.0 / (cce(config.ga_labels, predictions).numpy() + 0.00000001)
# Return the fitness
return solution_fitness
#==============================================================================
# Used for debugging, tells the progress of the GA
def callback_generation(ga_instance):
print("Generation = {generation}".format(generation=ga_instance.generations_completed))
print("Fitness = {fitness}".format(fitness=ga_instance.best_solution()[1]))
#==============================================================================
# Genetic Algorithm for optimization of classifier parameters
def ga(num_solutions, num_generations, num_parents_mating):
# Which model to optimize and the number of soultions in the population
keras_ga = pygad.kerasga.KerasGA(model=config.model, num_solutions=num_solutions)
# Cannot have more parents than solutions in the population
if(num_parents_mating > num_solutions):
print(f"Number of parents mating was set to", {num_solutions}, "(down from", {num_parents_mating}, "), because number of parents mating must be less than number of soultions in the population")
num_parents_mating = num_solutions
# Create the initial population
initial_population = keras_ga.population_weights
# Number of generations and parents mating in the GA
# Initial population of the GA
# Which fitness function the GA should use
ga_instance = pygad.GA(num_generations=num_generations, num_parents_mating=num_parents_mating, initial_population=initial_population, fitness_func=fitness_func, on_generation=callback_generation)
# Run the GA
ga_instance.run()
# Get best solution and parameters
solution, solution_fitness, solution_idx = ga_instance.best_solution()
best_solution_weights = pygad.kerasga.model_weights_as_matrix(model=config.model, weights_vector=solution)
# Return the best parameters found
return best_solution_weights
#==============================================================================
The following is the predictions and labels (in that order) when the error occured after 194 generations:
array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.]])
array([[1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
1.6537365e-21, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
1.3163597e-26, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
9.7521068e-33, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.6338916e-08, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9997479e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.5995245e-03, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 6.5632437e-15, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 5.8124778e-28,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 4.5765560e-22, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0761170e-25,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 8.3525886e-12, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 3.5442305e-20, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 5.3136361e-08,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.6981202e-15, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 2.0167827e-38,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 9.9991322e-01, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 5.0599866e-02, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.1111567e-27, 1.0000000e+00, 4.1279522e-01, 1.0000000e+00,
1.0000000e+00, 1.3650597e-17, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4416973e-23, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 7.1864803e-09,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 8.9365212e-32, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 3.1516618e-38],
[0.0000000e+00, 7.4043003e-28, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 1.5207636e-37, 5.2818956e-35],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.0324602e-37, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[2.0097023e-15, 9.9998254e-01, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 2.5984167e-12, 1.0000000e+00, 0.0000000e+00,
1.4740964e-32, 4.5718017e-33, 0.0000000e+00],
[1.6984066e-36, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.6022282e-08, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 4.3199220e-12, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9999988e-01, 1.0000000e+00, 1.1980822e-21,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
[0.0000000e+00, 7.8818629e-29, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.5401451e-16, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0748938e-07, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 9.9751586e-01, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.1370505e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.6451079e-16, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.1492193e-32,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 2.3176689e-28, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.5236378e-06, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9994671e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 3.8478177e-25, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 2.5574896e-22, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 9.9997914e-01, 1.0000000e+00,
0.0000000e+00, 5.1517299e-36, 6.5897658e-36],
[0.0000000e+00, 2.7718766e-28, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.0999998e-23, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 9.9995071e-01,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.4310640e-36, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 6.3025513e-10, 1.0000000e+00, 9.9999994e-01,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 2.3318755e-21, 9.4422745e-08],
[0.0000000e+00, 1.1929870e-15, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 1.8064725e-25, 1.4417658e-13],
[0.0000000e+00, 9.9369687e-01, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 3.2406049e-08, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
1.6537365e-21, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
1.6537365e-21, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395440e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971536e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395440e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971536e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 7.4395859e-03, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.6971518e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[1.0000000e+00, 0.0000000e+00, 1.0000000e+00, 0.0000000e+00,
1.3163597e-26, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00,
9.7521068e-33, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 9.9991322e-01, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 5.0599866e-02, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[2.0097023e-15, 9.9998254e-01, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 2.5984167e-12, 1.0000000e+00, 0.0000000e+00,
1.4740964e-32, 4.5718017e-33, 0.0000000e+00],
[0.0000000e+00, 1.5236378e-06, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9994671e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
[0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
[0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
[0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
[0.0000000e+00, 1.5236378e-06, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9994671e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.5236378e-06, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9994671e-01, 1.0000000e+00, 0.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.3270545e-13, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 9.9999976e-01, 9.9999970e-01, 1.8867243e-11,
0.0000000e+00, 7.4753842e-38, 0.0000000e+00],
[0.0000000e+00, 2.5574896e-22, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 9.9997914e-01, 1.0000000e+00,
0.0000000e+00, 5.1517299e-36, 6.5897658e-36],
[0.0000000e+00, 1.1929870e-15, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 1.8064725e-25, 1.4417658e-13],
[0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.1929870e-15, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 1.8064725e-25, 1.4417658e-13],
[0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00],
[0.0000000e+00, 1.1929870e-15, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 1.8064725e-25, 1.4417658e-13],
[0.0000000e+00, 1.1929825e-15, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 1.8064863e-25, 1.4417685e-13],
[0.0000000e+00, 2.1104829e-32, 1.0000000e+00, 1.0000000e+00,
1.0000000e+00, 1.0000000e+00, 1.0000000e+00, 1.0000000e+00,
0.0000000e+00, 0.0000000e+00, 0.0000000e+00]], dtype=float32)
I had the same issue of receiving NaN in the prediction when using a CNN with batch normalization and dropout. After removing that, the issue was resolved.