Hyper Parameter Tuning

Parameter searching:

• grid search

• random search

• Bayesian Optimization: SigOpt is providing parameter tuning service based on Bayesian optimization. This is a good place to start.

  • hyperopt (tree parzen estimator)
  • BayesianOptimization, Spearmint, MOE (Gaussian process)

  • SMAC (random forest regression)

    related papers: [1] [2] [3] [4]

• Spectral Optimization: Harmonica [github] currently only supports tuning binary parameters.

Model ensembling:

• different initialization
• different epochs
• different hyperparameters

Resources

• A toolkit sealing all types of parameter tuning methods: Google Vizier

• Tuning playbook