Is grid search method worth for neural networks?

Question

I have tried successfully to apply GridSearch Method to find the best parameters of a SVM and now I would like to apply it to neural networks (MLPClassifier)to find the best architecture (i.e. number of layers and of neurons/layer), the best activation function,...

However, I was wondering if this was not going to be too computationnally costly ? Moreover, how should I define my "grid" for the number of layers and of neurons ? Has this already been tested ? Thanks

Answer 1

Yes, a number of layers and neurons considered to be a part of hyper parameters.

In my opinion, grid search is not the best option for neural networks because of curse of dimensionality. People usually consider simple random search, especially at the earlier stages of the development. It works because quite often some hyper parameters do not affect loss function a lot, so there is no point in exhaustive searching through all possible combinations.

So, I would suggest the following loop:

1. Determine reasonable set of hyperparameters values (lower/upper bounds, steps)
2. Start random sampling for different combinations
3. After some time you will be able (hopefully) to narrow search space
4. Repeat with new bounds

In some special cases, you can use gradient based and Bayesian optimizations, but these options are very problem specific.