TY - JOUR
T1 - Gradient Descent Effects on Differential Neural Architecture Search
T2 - A Survey
AU - Santra, Santanu
AU - Hsieh, Jun Wei
AU - Lin, Chi Fang
N1 - Publisher Copyright:
© 2013 IEEE.
PY - 2021
Y1 - 2021
N2 - Gradient Descent, an effective way to search for the local minimum of a function, can minimize training and validation loss of neural architectures and also be incited in an appropriate order to decrease the searching cost of neural architecture search. In recent trends, the neural architecture search (NAS) is enormously used to construct an automatic architecture for a specific task. Mostly well-performed neural architecture search methods have adopted reinforcement learning, evolutionary algorithms, or gradient descent algorithms to find the best-performing candidate architecture. Among these methods, gradient descent-based architecture search approaches outperform all other methods in terms of efficiency, simplicity, computational cost, and validation error. In view of this, an in-depth survey is necessary to cover the usefulness of gradient descent method and how this can benefit neural architecture search. We begin our survey with basic concepts of neural architecture search, gradient descent, and their unique properties. Our survey then delves into the impact of gradient descent method on NAS and explores the effect of gradient descent in the search process to generate the candidate architecture. At the same time, our survey reviews mostly used gradient-based search approaches in NAS. Finally, we provide the current research challenges and open problems in the NAS-based approaches, which need to be addressed in future research.
AB - Gradient Descent, an effective way to search for the local minimum of a function, can minimize training and validation loss of neural architectures and also be incited in an appropriate order to decrease the searching cost of neural architecture search. In recent trends, the neural architecture search (NAS) is enormously used to construct an automatic architecture for a specific task. Mostly well-performed neural architecture search methods have adopted reinforcement learning, evolutionary algorithms, or gradient descent algorithms to find the best-performing candidate architecture. Among these methods, gradient descent-based architecture search approaches outperform all other methods in terms of efficiency, simplicity, computational cost, and validation error. In view of this, an in-depth survey is necessary to cover the usefulness of gradient descent method and how this can benefit neural architecture search. We begin our survey with basic concepts of neural architecture search, gradient descent, and their unique properties. Our survey then delves into the impact of gradient descent method on NAS and explores the effect of gradient descent in the search process to generate the candidate architecture. At the same time, our survey reviews mostly used gradient-based search approaches in NAS. Finally, we provide the current research challenges and open problems in the NAS-based approaches, which need to be addressed in future research.
KW - Gradient descent
KW - back-propagation
KW - evolutionary algorithm
KW - neural architecture search
KW - reinforcement learning
UR - http://www.scopus.com/inward/record.url?scp=85117585769&partnerID=8YFLogxK
U2 - 10.1109/ACCESS.2021.3090918
DO - 10.1109/ACCESS.2021.3090918
M3 - Article
AN - SCOPUS:85117585769
SN - 2169-3536
VL - 9
SP - 89602
EP - 89618
JO - IEEE Access
JF - IEEE Access
M1 - 9461192
ER -