In this study, a texture mask (TM) machine learning method for predicting metal surface roughness produced by different machining methods is proposed. The problems caused by angle deviation in the image acquisition process can be effectively improved, and the training time of the model can be reduced. The surface roughness, with a roughness average (Ra) below 1 um, produced by two similar processing methods, flat lapping and grinding, is examined for prediction and verification. The performances of TM and other feature extraction methods, under different irradiation system conditions and different angle deviations, are also evaluated and compared. The results show that the proposed TM method is more accurate than other methods when the problem of angle deviation occurs. We also compare TM with the convolutional neural network (CNN) method. The accuracy of both methods exceeds 91%, but the training time for TM is significantly less than that of the CNN method. The results show the texture mask method to be an accurate and efficient texture extraction method suitable for use in an automatic system.