Combined Bayesian and RNN-Based Hyperparameter Optimization for Efficient Model Selection Applied for autoML

The field of hyperparameter optimization (HPO) in auto machine learning (autoML) has been intensively studied, mainly in auto model selection (AMS), which finds the best set of hyperparameters, and neural architecture search (NAS), which optimizes the architecture of deep learning networks. In HPO, the two most significant problems are the demand of high level computational resources and the need of enormous computational time (GPU hours). In particular, the computational resources spent on HPO for complex deep learning networks are extremely high. Therefore, this paper augments HPO by adding recurrent neural networks (RNNs) to traditional statistical model-based algorithms to reduce the number of iterations of statistical models and eventually achieve the goal of lowering required computational resources. This paper’s main contribution is combining traditional statistical model-based algorithms and recurrent neural network models to reduce the computational time when doing HPO with deep learning.