I am using keras tuner in a Deep Learning project to fit hyperparameters.
Because there are many combinations of hyperparameters, even with a GPU it takes more than 24 hours to run. To have reproductible results, we fixed the random state (seed of numpy and tensorflow) before running the training.
I would like to know if there are an easy way to divide the tuning search into several steps so that we can run it each night (e.g divide it into 3 parts which takes each 8 hours to run). Thus, if there are 300 trials, we would like to run the first 100 trials the first night, trials 100 to 200 the second night, and finally the last 100 trials the last night.
I have try to search how to have the configuration of a particular trial_id (with the function populate_state of kera_tuner.tuners.random_search) so that I run a training on that configuration but I didn't find out how to use it.
Thanks in advance for your answers.
maybe I did not understood the question, but if problem is time comsumption , why not use hyperband instead of random search ?
EDIT : also there is a chapter in link about distributed tuning . Or you could split your hyperparameters list into 3 or 4 HyperParameters object and run them one per night (like the example in link