Is there a good framework or package in python to compare something over a number of parameters?

Question

When working in python for research I often end up with some problem where I want to compare a result over a different number of parameters.

params1 = [0.1, 0.2, 0.3]
params2 = [5, 10, 50]
parmas3 = range(8)

for (p1, p2, p3) in itertools.product(params1, params2, params3):
    result = evaluate(p1, p2, p3)

Is there some good way to manage a setup like this for many parameters, where I have a good way to

• save the results
• plot and analyse them conveniently
• probably running multiple experiments in parallel
• maybe even some caching and saving of intermediate results (e.g. if you want to continue the run later

I was thinking that this must be a pretty common pattern, but I was not able to find something that even slightly goes into this direction.

Answer 1

If you build models using the sci-kit learning library, then sci-kit has an internal package called model_selection. specially, GridSearchCV good for your case.

• scikit-optimize
• hyperopt
• Spearmint are popular libraries for hyperparameter optimization.