GA for hyper parameter tuning for RandomForestRegressor

Question

I wrote a genetic algorithm (at least, I think it is a GA) that attempts to tune hyper parameters for the kaggle.com Housing Prices Competition for Kaggle Learn Users.

I am new to GA so I'd like to get an experienced opinion about how well is my algorithm performing. In the current run it managed to fit a model and get MAE of 16308.82 after 15 generations and about 6 hours. It has been running for 10 more hours now and didn't get better results so I guess this is it for this version (there may be ways to improve the algorithm of course). Is this good performance? can it be improved?

I keep the code in jupyter notebooks on github: https://github.com/ZackYovel/GA-for-Hyper-Parameter-Tuning. Sorry for the horrible code, I like to start quick and dirty and refactor later. I am currently working on a library that will enable me to implement the algorithm using cleaner code.

The algorithm I used in my current run is:

1. Produce 150 individuals (individual=set of hyper parameter values) divided into 5 territories (populations)
2. Every individual has in addition to the hyper parameter values also a small_step_mutation and a large_step_mutation values (explained ahead).
3. Evaluate all individuals and keep the 15 best in each population
4. Crossover: the best individual in each population is paired in a loop with all other survivors. For each pair, each hyper parameter is copied from a randomly chosen parent (e.g. for each hp: choose parent = mom or dad randomly, then copy parent[hp] to child)
5. Go back to step 3.

small_step_mutation determines the standard deviation used to change the hyper parameter value when the current value is the mean (e.g. new_hp_value = np.random.normal(current_hp_value, small_step_mutation * (max_value_for_hp - min_value_for_hp))

large_step_mutation is the probability that a large mutation will occur (if -choose True or False with probability large_step_mutation for True- then: new_hp_value = random(min_value, max_value)

The two mutation parameters are under evolution. small_step_mutation tends to evolve to zero, large_step_mutation tends to evolve to 1.

Steps 3-5 are repeated in a loop of 500 generations but I never let it actually run the entire 500 generations because each generation can take a pretty long time (varies between 15 minutes to several hours per generation. I consider this a bug and I think I know how to fix it. Will try soon. I expect the average runtime per generation to be 15-30 minutes)

So bottom line I have two questions:

1. Is my current performance good? can it be improved?
2. Is my algorithm good? can it be improved? (am I actually doing GA or something else?)

Thanks in advance!

Answer 1

The steps of GA is below:

START
Generate the initial population
Compute fitness
REPEAT
    Selection
    Crossover
    Mutation
    Compute fitness
UNTIL population has converged
STOP

1. Is my current performance good? can it be improved? If the result of your task is increased, then you did it well. Else you need to find out some improvement of GA or new Evolutionary algorithms such as "Differential evolution" and "Particle swarm optimization".
2. Is my algorithm good? can it be improved? (am I actually doing GA or something else?) You need some change into your code to improve its performance. First, the mutation should lie in the loop to increase exploration power of your algorithm. Second, in cross over operation, you should divide the population into pairs of solution and then apply cross over on all pairs. It makes a new diverse population. Third, in each iteration, the best individual should be stored (elitism).