Why are the mean_q and mae for keras-rl2 DQN agent logged as NaN

Question

Copied the codes over from https://github.com/keras-rl/keras-rl/blob/master/examples/dqn_atari.py but only the rewards and number of steps are logged and the error metrics are all NaN

memory = SequentialMemory(limit=1000000, window_length=WINDOW_LENGTH)

processor = AtariProcessor()

policy = LinearAnnealedPolicy(EpsGreedyQPolicy(), attr='eps', value_max=1., value_min=.1, value_test=.05,nb_steps=1000000)

dqn = DQNAgent(model=model1, nb_actions=nb_actions, policy=policy, memory=memory,
           processor=processor, nb_steps_warmup=50000, gamma=.99, 
target_model_update=10000,train_interval=4, delta_clip=1.)

adamOptimizer = adam_v2.Adam(learning_rate=0.00025)

dqn.compile(adamOptimizer ,metrics=['mae'])

env_name = 'PongNoFrameskip-v4'

weights_filename = 'dqn_{}_weights.h5f'.format(env_name)

checkpoint_weights_filename = 'dqn_' + env_name + '_weights_{step}.h5f'

log_filename = 'dqn_{}_log.json'.format(env_name)

callbacks = [ModelIntervalCheckpoint(checkpoint_weights_filename, interval=250000)]

callbacks += [FileLogger(log_filename, interval=100)]

trainLog = dqn.fit(env, callbacks=callbacks, nb_steps=1750000, log_interval=10000)    

I only let it train for a few thousand steps just for show, and in the dqn_{}.log.json file the mean_q , the loss and the mae are all NaN, below is a screenshot of the json log file content

dqn agent training log file

and when the callbacks history keys are printed, loss and mae are not included

print(trainLog.history.keys())

output : dict_keys(['episode_reward', 'nb_episode_steps', 'nb_steps'])

Answer 1

They didn't implemented it (and probably won't since the library is now archive). However, I solve this by modifying the source code at keras-rl2/rl/core.py in line 219 or something, add the code i put in between the ########.

if done:
    # We are in a terminal state but the agent hasn't yet seen it. We therefore
    # perform one more forward-backward call and simply ignore the action before
    # resetting the environment. We need to pass in `terminal=False` here since
    # the *next* state, that is the state of the newly reset environment, is
    # always non-terminal by convention.
    self.forward(observation)
    self.backward(0., terminal=False)

    # This episode is finished, report and reset.

    episode_logs = {
        'episode_reward': episode_reward,
        'nb_episode_steps': episode_step,
        'nb_steps': self.step,
        #################################
        **{name:metrics[i] for i, name in enumerate(self.metrics_names)},
        'info': accumulated_info,
        #################################
    }
    callbacks.on_episode_end(episode, episode_logs)

    episode += 1
    observation = None
    episode_step = None
    episode_reward = None

I also added the info just in case. Don't worry, this won't modify the agents training process or behavior, we are just retrieving additional information.