The analysis of electroencephalography (EEG) recordings has attracted increasing interest in recent decades and provides the pivotal scientific tool for researchers to quantitatively study brain activity during sleep, and has extended our knowledge of the fundamental mechanisms of sleep physiology. Conventional EEG analyses are mostly based on Fourier transform technique which assumes linearity and stationarity of the signal being analyzed. However, due to the complex and dynamical characteristics of EEG, nonlinear approaches are more appropriate for assessing the intrinsic dynamics of EEG and exploring the physiological mechanisms of brain activity during sleep. Therefore, this article introduces the most commonly used nonlinear methods based on the concepts of fractals and entropy, and we review the novel findings from their clinical applications. We propose that nonlinear measures may provide extensive insights into brain activities during sleep. Further studies are proposed to mitigate the limitations and to expand the applications of nonlinear EEG analysis for a more comprehensive understanding of sleep dynamics.