The effect of Mg doping on the electrical and optical properties of the p-GaN/AlGaN structures on a Si substrate grown by metal organic chemical vapor deposition was investigated. The Hall measurement showed that the activation efficiency of the sample with a 450 sccm Cp2Mg flow rate reached a maximum value of 2.22%. No reversion of the hole concentration was observed due to the existence of stress in the designed sample structures. This is attributed to the higher Mg-to-Ga incorporation rate resulting from the restriction of self-compensation under compressive strain. In addition, by using an AlN interlayer (IL) at the interface of p-GaN/AlGaN, the activation rate can be further improved after the doping concentration reaches saturation, and the diffusion of Mg atoms can also be effectively suppressed. A high hole concentration of about 1.3 × 1018 cm−3 can be achieved in the p-GaN/AlN-IL/AlGaN structure.