Three-phase behavior-based detection and classification of known and unknown malware

To improve both accuracy and efficiency in detecting known and even unknown malware, we propose a three-phase behavior-based malware detection and classification approach, with a faster detector in the first phase to filter most samples, a slower detector in the second phase to observe remaining ambiguous samples, and then a classifier in the third phase to recognize their malware type. The faster detector executes programs in a sandbox to extract representative behaviors fed into a trained artificial neural network to evaluate their maliciousness, whereas the slower detector extracts and matches the LCSs of system call sequences fed into a trained Bayesian model to calculate their maliciousness. In the third phase, we define malware behavior vectors and calculate the cosine similarity to classify the malware. The experimental results show that the hybrid two-phase detection scheme outperforms the one-phase schemes and achieves 3.6% in false negative and 6.8% in false positive. The third-phase classifier also distinguishes the known-type malware with an accuracy of 85.8%.