The solar cell has a poor spectral response in the UV region, which affects its power conversion efficiency (PCE). The utilization of a luminescent downshifting (LDS) layer has been suggested to improve the spectral response of the photovoltaics in the short wavelength region through photoluminescence (PL) conversion and antireflection effects, which then enhance the PCE of the solar cell. Recently, colloidal quantum dots (CQDs) or perovskite quantum dots (PQDs) have been gaining prime importance as an LDS material due to their eminent optical characteristics, such as their wide absorption band, adjustable visible emission, short PL lifetime, and near-unity quantum yields. However, the instability of QDs that occurs under certain air, heat, and moisture conditions limits its commercialization. Thus, in this review, we will focus on the physical and optical characteristics of QDs. Further, we will discuss different synthesis approaches and the stability issues of QDs. Different approaches to improve the stability of QDs will be discussed in detail alongside the recent breakthroughs in QD-based solar cells for various applications and their current challenges. We expect that this review will provide an effective gateway for researchers to fabricate LDS-layer-based solar cells.