Background: The alcohol patch test (APT) can detect aldehyde dehydrogenase (ALDH) genetic polymorphisms used to diagnose cutaneous erythema. However, the subjective results can vary owing to confounding factors. The hue-saturation-value (HSV) model provides an objective means of image analysis with APT. Methods: This study enrolled 57 participants (27.7 ± 9.0 years, 52.6% females) with ALDH2*1/*1, ALDH2*1/*2, and ALDH2*2/*2 percentages of 50.9%, 43.8%, and 5.3%, respectively. In total, 56 APT protocols were applied and analyzed employing both visual inspection and the HSV model. The value of the delta standard deviation (SD) of the hue histogram, which manifests the difference between the APT reaction and the baseline skin color, was obtained using the HSV model. The receiver operating characteristic (ROC) curve and area under the ROC curve (AUC) were used to predict the ALDH2*2 allele with the HSV model. Results: Upon visual inspection, a maximal Youden index with a sensitivity of 82.1% and a specificity of 96.6% was determined for the ALDH2 genetic mutation. Using the delta SD of hue obtained in the HSV model, a maximal Youden index with 85.7% sensitivity and 96.6% specificity was determined using the ROC curve analysis (AUC = 0.948, p < 0.001). Thus, the use of the HSV model analysis with APT resulted in equal specificity, but better sensitivity, compared to those obtained upon visual inspection. Conclusion: The HSV model took into account the potential confounding factors, and thus, could help in the prediction of ALDH2 genetic polymorphisms.