In this paper, an approach is proposed to analyze and compensate geometric errors of five-axis machine tools using a novel optical measurement instrument. The instrument consists of two pairs of QDs (quadrant detectors) and triangulation laser probes, one magnetic base and one glass ball. The kinematic and geometric error models of a five-axis machine with a titling rotary table are obtained using the HTM (homogeneous transformation matrix). The error model totally includes 43 geometric errors. The proposed method first analyzes the sensitivity Jacobian matrix and observes what are the critical offsets of five-axis machine which dominant the overall volumetric errors. Three types of error sources consists of the center bias, circle and eccentric are identified. NC test paths such as BK2 are planned according to the inverse kinematic equations. Simulations are performed to generalize the polar plots of error patterns. Finally, the proposed method is applied to compensate the geometric errors of the five-axis machine tool, and the machine accuracy can be significantly improved.