This study introduces a tool wear monitoring system that uses multiple sensors and a feature fusion technique. To improve the robustness of the system, different tightening torque and spindle speed conditions were considered in the experimental design stage. Vibration signals in three coordinates and sound signals were collected and transformed by fast Fourier transform for feature extraction. Two types of features in the frequency domain were used to establish the proposed system, including the mean value features selected by the class mean scatter feature selection criterion and statistical features by Gradient Class Activation Mapping++ (Grad-CAM++). The proposed monitoring system was established using a hierarchical neural network structure and feature fusion of the sensors. Cross-validation was introduced to demonstrate the performance and effectiveness of our approach under various tightening torque values and spindle speeds.