Gradient Descent Effects on Differential Neural Architecture Search: A Survey

Santanu Santra, Jun Wei Hsieh*, Chi Fang Lin

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

25 Scopus citations

Abstract

Gradient Descent, an effective way to search for the local minimum of a function, can minimize training and validation loss of neural architectures and also be incited in an appropriate order to decrease the searching cost of neural architecture search. In recent trends, the neural architecture search (NAS) is enormously used to construct an automatic architecture for a specific task. Mostly well-performed neural architecture search methods have adopted reinforcement learning, evolutionary algorithms, or gradient descent algorithms to find the best-performing candidate architecture. Among these methods, gradient descent-based architecture search approaches outperform all other methods in terms of efficiency, simplicity, computational cost, and validation error. In view of this, an in-depth survey is necessary to cover the usefulness of gradient descent method and how this can benefit neural architecture search. We begin our survey with basic concepts of neural architecture search, gradient descent, and their unique properties. Our survey then delves into the impact of gradient descent method on NAS and explores the effect of gradient descent in the search process to generate the candidate architecture. At the same time, our survey reviews mostly used gradient-based search approaches in NAS. Finally, we provide the current research challenges and open problems in the NAS-based approaches, which need to be addressed in future research.

Original languageEnglish
Article number9461192
Pages (from-to)89602-89618
Number of pages17
JournalIEEE Access
Volume9
DOIs
StatePublished - 2021

Keywords

  • Gradient descent
  • back-propagation
  • evolutionary algorithm
  • neural architecture search
  • reinforcement learning

Fingerprint

Dive into the research topics of 'Gradient Descent Effects on Differential Neural Architecture Search: A Survey'. Together they form a unique fingerprint.

Cite this